Introduction
We are going to turn on three CoreOS VMs under vagrant and set them under various configurations to show off different failure domains of Kubernetes and how to handle them in production.
git clone https://github.com/coreos/coreos-vagrant
cd coreos-vagrant
git clone https://github.com/philips/real-world-kubernetes
sed -e 's%num_instances=1%num_instances=3%g' < config.rb.sample > config.rb
NOTE: please use the latest version of CoreOS alpha box
vagrant box update --box coreos-alpha
vagrant box update --box coreos-alpha --provider vmware_fusion
Now lets startup the hosts
vagrant up
vagrant status
And configure ssh to talk to the new vagrant hosts correctly:
vagrant ssh-config > ssh-config
alias ssh="ssh -F ssh-config"
alias scp="scp -F ssh-config"
This should show three healthy CoreOS hosts launched.
etcd clustering
For etcd we are going to scale the cluster from a single machine up to a three machine cluster. Then we will fail a machine and show everything is still working.
Single Machine
Setup an etcd cluster with a single machine on core-01. This is as easy as starting the etcd2 service on CoreOS.
vagrant up
vagrant ssh-config > ssh-config
vagrant ssh core-01
sudo systemctl start etcd2
systemctl status etcd2
Confirm that you can write into etcd now:
etcdctl set kubernetes rocks
etcdctl get kubernetes
Now, we can confirm the cluster configuration with the etcdctl member list subcommand.
etcdctl member list
By default etcd will listen on localhost and not advertise a public address. We need to fix this before adding additional members. First, tell the cluster the members new address. The IP is the default IP for core-01 in coreos-vagrant.
etcdctl member update ce2a822cea30bfca http://172.17.8.101:2380
Updated member with ID ce2a822cea30bfca in cluster
Let's reconfigure etcd to listen on public ports to get it ready to cluster.
sudo su
sudo mkdir /etc/systemd/system/etcd2.service.d/
cat <<EOM > /etc/systemd/system/etcd2.service.d/10-listen.conf
[Service]
Environment=ETCD_NAME=core-01
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.101:2379
Environment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Environment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
EOM
All thats left is to restart etcd and the reconfiguration should be complete.
sudo systemctl daemon-reload
sudo systemctl restart etcd2
etcdctl get kubernetes
Add core-02 to the Cluster
Now that core-01 is ready for clustering lets add our first additional cluster member, core-02.
vagrant ssh core-02
etcdctl --peers http://172.17.8.101:2379 set /foobar baz
Login to core-02 and lets add it to the cluster:
etcdctl --peers http://172.17.8.101:2379 member add core-02 http://172.17.8.102:2380
The above command will dump out a bunch of initial configuration information. Next, we will put that configuration information into the systemd unit file for this member:
sudo su
mkdir /etc/systemd/system/etcd2.service.d/
cat <<EOM > /etc/systemd/system/etcd2.service.d/10-listen.conf
[Service]
Environment=ETCD_NAME=core-02
Environment=ETCD_INITIAL_CLUSTER=core-01=http://172.17.8.101:2380,core-02=http://172.17.8.102:2380
Environment=ETCD_INITIAL_CLUSTER_STATE=existing
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.102:2379
Environment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Environment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
EOM
sudo systemctl daemon-reload
sudo systemctl restart etcd2
etcdctl member list
Now, at this point the cluster is in an unsafe configuration. If either machine fails etcd will stop working.
sudo systemctl stop etcd2
exit
vagrant ssh core-01
sudo etcdctl set kubernetes bad
sudo etcdctl get kubernetes
exit
vagrant ssh core-02
sudo systemctl start etcd2
sudo etcdctl set kubernetes awesome
Add core-03 to the Cluster
To get out of this unsafe configuration lets add a third member. After adding this third member this cluster will be able to survive single machine failures.
vagrant ssh core-03
etcdctl --peers http://172.17.8.101:2379 member add core-03 http://172.17.8.103:2380
sudo su
mkdir /etc/systemd/system/etcd2.service.d/
cat <<EOM > /etc/systemd/system/etcd2.service.d/10-listen.conf
[Service]
Environment=ETCD_NAME=core-03
Environment=ETCD_INITIAL_CLUSTER=core-01=http://172.17.8.101:2380,core-02=http://172.17.8.102:2380,core-03=http://172.17.8.103:2380
Environment=ETCD_INITIAL_CLUSTER_STATE=existing
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://172.17.8.103:2379
Environment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Environment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
EOM
sudo systemctl daemon-reload
sudo systemctl restart etcd2
etcdctl member list
Surviving Machine Failure
Now we can have a single machine fail, like core-01, and have the cluster configure to set and retrieve values.
vagrant destroy core-01
vagrant ssh core-02
etcdctl set foobar asdf
Automatic Bootstrapping
This exercise was designed to get you comfortable with etcd bringup and reconfiguration. In environments where you have deterministic IP address you can use static cluster bringup. In environments with dynamic IPs you can use an etcd discovery.
Cleanup
For the next exercise we are only going to use a single member etcd cluster. Lets destroy the machines and bring up clean hosts:
vagrant destroy -f core-01
vagrant destroy -f core-02
vagrant destroy -f core-03
Disaster Recovery of etcd
In most all environments etcd will be replicated. But, etcd is generally holding onto critical data so you should plan for backups and disaster recovery. This example will cover restoring etcd from backup.
Start a Cluster and Destroy
Bring up the cluster of three machines
vagrant up
vagrant ssh-config > ssh-config
Startup a single machine etcd cluster on core-01 and launch a process that will write a key now every 5 seconds with the current date.
ssh core-02
systemctl start etcd2
sudo systemd-run /bin/sh -c 'while true; do etcdctl set now "$(date)"; sleep 5; done'
exit
Backup the etcd cluster state to a tar file and save it on the local filesystem. In a production cluster this could be done with a tool like rclone in a container to save it to an object store or another server.
ssh core-02 sudo tar cfz - /var/lib/etcd2 > backup.tar.gz
ssh core-02 etcdctl get now
vagrant destroy -f core-02
vagrant up core-02
vagrant ssh-config > ssh-config
Restore from Backup
First, lets restore the data from the etcd member to the new host location:
scp backup.tar.gz core-01:
ssh core-01
tar xzvf backup.tar.gz
sudo su
mv var/lib/etcd2/member /var/lib/etcd2/
chown -R etcd /var/lib/etcd2
Next we need to tell etcd to start but to only use the data, not the cluster configuration. We do this by setting a flag called FORCE_NEW_CLUSTER. This is something like "single user mode" on a Linux host.
mkdir -p /run/systemd/system/etcd2.service.d
cat <<EOM > /run/systemd/system/etcd2.service.d/10-new-cluster.conf
[Service]
Environment=ETCD_FORCE_NEW_CLUSTER=1
EOM
systemctl daemon-reload
systemctl restart etcd2
To ensure we don't accidently reset the cluster configuration in the future, remove the force new cluster option and flush it from systemd.
rm /run/systemd/system/etcd2.service.d/10-new-cluster.conf
systemctl daemon-reload
Now, we should have our database fully recovered with the application data intact. From here we can rebuild the cluster using the methods from the first section.
etcdctl member list
etcdctl get now
Cleanup
vagrant destroy -f core-01
vagrant destroy -f core-02
vagrant destroy -f core-03
Securing etcd
Now that we have good practice with cluster operations of etcd under network partition, adding/removing members, and backups lets add transport security to the machine that will act as our etcd machine: core-01.
vagrant up
vagrant ssh-config > ssh-config
Generate Certificate Authority
First, lets generate a certificate authority and some certificates signed by that authority. You can take a look at the makefile but it essentially using the cfssl tool to generate a CA and an etcd cert signed by that CA.
pushd real-world-kubernetes/tls-setup
make install-cfssl
make
popd
Now drop the certs onto the host:
scp -r real-world-kubernetes/tls-setup/certs core-01:
Use CA with etcd
Install the newly generated certificates onto the host. In a real-world environment this would be done with a cloud-config or installed on first boot.
ssh core-01
sudo su
mkdir /etc/etcd
mv certs/etcd* /etc/etcd
chown -R etcd: /etc/etcd
cp certs/ca.pem /etc/ssl/certs/
/usr/bin/c_rehash
exit
exit
Finally, prepare etcd to use a certificate and key file that are dropped onto the host.
ssh core-01
sudo su
mkdir /etc/systemd/system/etcd2.service.d/
cat <<EOM > /etc/systemd/system/etcd2.service.d/10-listen.conf
[Service]
Environment=ETCD_NAME=core-01
Environment=ETCD_ADVERTISE_CLIENT_URLS=https://core-01:2379
Environment=ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
Environment=ETCD_CERT_FILE=/etc/etcd/etcd.pem
Environment=ETCD_KEY_FILE=/etc/etcd/etcd-key.pem
EOM
systemctl daemon-reload
systemctl restart etcd2
exit
exit
Test with etcdctl
With everything in place we should be able to set a key over a secure connection:
ssh core-01
etcdctl --peers https://core-01:2379 --ca-file certs/ca.pem set kubernetes is-ready
Running Kubernetes API Server
Now that we fully understand etcd and how to operate it securely and in clusters lets bringup a Kubernetes API server.
CONTROLLER=core-01
Get the basic configuration files in place on the server.
scp -r real-world-kubernetes/k8s-setup ${CONTROLLER}:
scp -r real-world-kubernetes/k8s-srv-setup ${CONTROLLER}:
Then copy them over to the right locations and restart the kubelet to have it bootstrap the API server.
ssh ${CONTROLLER}
sudo su
mkdir -p /etc/kubernetes/ssl/
cp certs/ca.pem /etc/kubernetes/ssl/
cp certs/apiserver* /etc/kubernetes/ssl/
cp k8s-setup/kubelet.service /etc/systemd/system/kubelet.service
mkdir -p /etc/kubernetes/manifests/
cp k8s-setup/kube-*.yaml /etc/kubernetes/manifests/
mkdir -p /srv/kubernetes/manifests
cp k8s-srv-setup/*.yaml /srv/kubernetes/manifests
systemctl daemon-reload
systemctl restart kubelet.service
systemctl enable kubelet
exit
exit
Test API
At this point the API should be up and available. But, we need a DNS entry to point at; so lets set that up first on our workstations. NOTE: this IP will change based on host configuration.
export CORE_01_IP=$(cat ssh-config | grep HostName | awk '{print $2}' | head -n1)
sudo -E /bin/sh -c 'echo "${CORE_01_IP} core-01" >> /etc/hosts'
With DNS configured we can try kubectl with our pre-made configuration file:
export KUBECONFIG=real-world-kubernetes/kubeconfig
kubectl get pods
If all goes well we shoudl get an empty list of pods! Now, lets add some worker nodes.
Kubernetes API Server Under etcd Failure
Temporary Partition
Lets start a really boring job on the cluster that just sleeps forever. This goes through just fine:
kubectl run pause --image=gcr.io/google_containers/pause
Next, we will stop etcd simulating a partition:
ssh core-01 sudo systemctl stop etcd2
Attempting to do any API call to the server is going to fail blocked on etcd. This behavior is identical to if you had a web service and stopped its SQL database.
kubectl describe rc pause
Lets start up etcd and get things going:
ssh core-01 sudo systemctl start etcd2
After a few seconds the API server should start responding and we should be able to get the status of our replication controller:
kubectl describe rc pause
Data-loss and Restore
Lets run a really boring application in this cluster with no nodes:
kubectl run pause --image=gcr.io/google_containers/pause
And take a quick backup of etcd:
ssh core-01 sudo tar cfz - /var/lib/etcd2 > backup.tar.gz
kubectl scale rc pause --replicas=5
kubectl describe rc pause
ssh core-01 sudo systemctl stop etcd2
ssh core-01
sudo su
mkdir tmp
mv /etc/kubernetes/manifests/kube-* tmp/
rm -Rf /var/lib/etcd2/*
exit
docker ps
exit
scp backup.tar.gz core-01:
ssh core-01
tar xzvf backup.tar.gz
sudo su
mv var/lib/etcd2/member /var/lib/etcd2/
chown -R etcd /var/lib/etcd2
systemctl start etcd2.service
mv tmp/* /etc/kubernetes/manifests
exit
etcdctl --peers https://core-01:2379 --ca-file certs/ca.pem set kubernetes is-ready
exit
kubectl describe rc pause
kubectl scale rc pause --replicas=1
Kubernetes Workers
Setup Workers
Lets setup core-02 and core-03 as the worker machines.
WORKER=core-02
scp -r real-world-kubernetes/worker-setup ${WORKER}:
scp -r real-world-kubernetes/tls-setup/certs ${WORKER}:
ssh ${WORKER}
sudo su
mkdir -p /etc/kubernetes/ssl/ /etc/kubernetes/manifests
cp certs/ca.pem /etc/kubernetes/ssl/
cp certs/worker* /etc/kubernetes/ssl/
cp worker-setup/kubelet.service /etc/systemd/system/kubelet.service
cp worker-setup/kube-*.yaml /etc/kubernetes/manifests/
cp worker-setup/worker-kubeconfig.yaml /etc/kubernetes
systemctl daemon-reload
systemctl restart kubelet.service
systemctl enable kubelet
exit
exit
Now re-run the above after changing the worker variable to setup core-03:
WORKER=core-03
At this point we should see two machines listed in the set of machines
kubectl get nodes
Individual Worker Failure
kubectl describe rc pause
kubectl scale rc pause --replicas=10
vagrant halt core-01
Now after about one minute we should notice everything has been moved off of core-03. Why? It is because of a thing called the
kubectl describe node core-03 core-02
High Availability of API Server
This is easy, the API server is trivially horizontally scalable.
Set the controller to core-02 and re-run the controller provisioning steps from before against this host.
CONTROLLER=core-02
Now test out that the kubernetes services are running.
kubectl -s https://core-02 get pods
High-Availability of Scheduler/Controller-Manager
There is a service called the pod master that moves things in and out of the kubelet's manifest directory based on compare and swap operations in etcd.
Let's force a master election of the scheduler by removing all of the control pieces, including the pod master out of the host.
ssh core-01
sudo su
mkdir tmp
mv /etc/kubernetes/manifests/* tmp
After a few seconds we should see that the scheduler and controller manager get master elected over to
ssh core-02
sudo su
ls -la /etc/kubernetes/manifests
With this mechanism in place and an HA etcd cluster you can rest easy knowing the control plane won't go down in the face of single machine failure.
Cleanup
Upgrading Control Cluster
Now upgrading under the control plane between minor versions is something you might realistically do. However, these sorts of scenarios aren't really tested.