Creating high availability in a Kubernetes environment is critical to ensuring robust and highly available applications in the event of a disaster or hardware failure. One solution to this problem is to use multiple control planes or masters nodes in a cluster.
In the previous post it has been explained to install a kubernetes cluster using containerd. this time we will use containerd as the runtime container because kubernetes support for docker is about to end.
The following is a list of servers used:
And the topology used as follows :
I will skip the containerd configuration, you can see how to configure containerd in here https://medium.com/@mrzik/how-to-create-kubernetes-cluster-with-containerd-90399ec3b810
Step 1. Configure /etc/hosts
Do this configuration on all master nodes.
Add the following list to hosts files so each node can communicate with names
root@kubmaster1:~# vi /etc/hosts
127.0.0.1 localhost
127.0.1.1 kubmaster1#Kubernetes Nodes
172.16.4.74 kubmaster1.mylab.local kubmaster1
172.16.4.75 kubmaster2.mylab.local kubmaster2
172.16.4.76 kubmaster3.mylab.local kubmaster3
172.16.4.77 kubworker1.mylab.local kubworker1
172.16.4.78 kubworker2.mylab.local kubworker2
172.16.4.79 kubworker3.mylab.local kubworker3#Keepalived Virtual IP
172.16.4.80 kubvip.mylab.local kubvip#Kubectl Client
172.16.4.81 kubclient.mylab.local kubclient# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
root@kubmaster1:~#
Step 2. Configure keepalived
Do this configuration on all master nodes
Install keepalived package
sudo apt -y install keepalived
Create check script for keepalived
sudo vi /etc/keepalived/check_apiserver.sh#!/bin/sh
errorExit() {
echo "*** $*" 1>&2
exit 1
}
curl --silent --max-time 2 --insecure https://localhost:8443/ -o /dev/null ||
errorExit "Error GET https://localhost:8443/"
if ip addr | grep -q 172.16.4.80; then
curl --silent --max-time 2 --insecure https://172.16.4.80:8443/ -o
/dev/null || errorExit "Error GET https://172.16.4.80:8443/"
fi
Add permisson for executing check script
sudo chmod +x /etc/keepalived/check_apiserver.sh
Create file configuration for keepalived
sudo vi /etc/keepalived/keepalived.conf#Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
enable_script_security
}
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 3
weight -2
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens160 #costumized this with your interface
virtual_router_id 100
priority 255 #change priority for other node
authentication {
auth_type PASS
auth_pass password #define your password
}
virtual_ipaddress {
172.16.4.80/24
}
track_script {
check_apiserver
}
}
Enable and start keepalived service
systemctl enable --now keepalived
systemctl status keepalived
Repeat this step for other nodes ( kubmaster2 and kubmaster3 ), for kubmaster2 and kubmaster3 there are some changes that need to be adjusted ( state to BACKUP , priority to 254 and 253)
Step 3. Configure haproxy
Do this configuration on all master nodes
Install haproxy package
sudo apt -y install haproxy
Add the following lines to the end of haproxy.cfg
sudo vi /etc/haproxy/haproxy.cfg#Kubernetes APIServer Frontend
frontend apiserver
bind 172.16.4.80:8443
mode tcp
option tcplog
default_backend apiserver#Kubernetes APIServer Backend
backend apiserver
option httpchk GET /healtz
http-check expect status 200
mode tcp
option tcp-check
balance roundrobin
server kubmaster1.mylab.local:6443 check
server kubmaster2.mylab.local:6443 check
server kubmaster3.mylab.local:6443 check
Start and enable haproxy service
sudo systemctl enable --now haproxy
sudo systemctl status haproxy
Repeat step for other master nodes
Step 4. Configure kubernetes requirement
Do this configuration on control plane/master nodes and worker nodes
Disable swap
sudo swapoff -a
sudo vi /etc/fstab#comment swap disk
#/swap.img none swap sw 0 0#save and exit :wq
Configure sysctl
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOFsudo sysctl --system
Allow required port for kubernetes or Disable Firewall
sudo ufw disable
Setup iptables backend to use iptables-legacy
sudo update-alternatives --config iptablesThere are 2 choices for the alternative iptables (providing /usr/sbin/iptables).Selection Path Priority Status
------------------------------------------------------------
0 /usr/sbin/iptables-nft 20 auto mode
* 1 /usr/sbin/iptables-legacy 10 manual mode
2 /usr/sbin/iptables-nft 20 manual modePress <enter> to keep the current choice[*], or type selection number:
Add kubernetes repository
sudo wget https://packages.cloud.google.com/apt/doc/apt-key.gpgsudo mv apt-key.gpg /etc/apt/trusted.gpg.d/sudo cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOFsudo apt update
sudo apt install -y kubeadm kubelet
Enable kubelet service but dont start yet
sudo systemctl enable kubelet
Step 5. Kubeadm Cluster Init
Do this configuration on kubmaster1
kubeadm init with external endpoint and use containerd as container runtime
sudo kubeadm init --control-plane-endpoint "kubvip.mylab.local:8443" --upload-certs --cri-socket /run/containerd/containerd.sock
Note final result of kubeadm init cause we will use it for join another control plane and worker
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root:kubeadm join kubvip.mylab.local:8443 --token x65nj4.l2vi5s9r8cmqofj5 \
--discovery-token-ca-cert-hash sha256:859cf17f33574ad60b67b3694b2348c2c27d292ca597a1f3d5dba84948ac2b6b \
--control-plane --certificate-key c999c6807438d200627b7d3b5d6e247441814949360226f6d5837dae4e6139a6Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:kubeadm join kubvip.mylab.local:8443 --token x65nj4.l2vi5s9r8cmqofj5 \
--discovery-token-ca-cert-hash sha256:859cf17f33574ad60b67b3694b2348c2c27d292ca597a1f3d5dba84948ac2b6b
On kubmaster2 and kubmaster3 run this command to joining cluster as control plane nodes and add containerd as container runtime
sudo kubeadm join kubvip.mylab.local:8443 --token x65nj4.l2vi5s9r8cmqofj5 \
--discovery-token-ca-cert-hash sha256:859cf17f33574ad60b67b3694b2348c2c27d292ca597a1f3d5dba84948ac2b6b \
--control-plane --certificate-key c999c6807438d200627b7d3b5d6e247441814949360226f6d5837dae4e6139a6 --cri-socket /run/containerd/containerd.sock
On kubworker1–3 run this command to joining cluster as worker nodes and add containerd as container runtime
sudo kubeadm join kubvip.mylab.local:8443 --token x65nj4.l2vi5s9r8cmqofj5 \
--discovery-token-ca-cert-hash sha256:859cf17f33574ad60b67b3694b2348c2c27d292ca597a1f3d5dba84948ac2b6b --cri-socket /run/containerd/containerd.sock
Step 6. Add Calico CNI
Do this configuration on kubclient node
Before add calico cni we need to install and configure kubectl on kubclient node
sudo wget https://packages.cloud.google.com/apt/doc/apt-key.gpgsudo mv apt-key.gpg /etc/apt/trusted.gpg.d/sudo cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOFsudo apt update
sudo apt -y install kubectl
Create .kube directory on home directory
mkdir -p $HOME/.kube
SSH to kubmaster1 and scp kubernetes admin.conf to kubclient as config file
sudo scp /etc/kubernetes/admin.conf youradminuser@kubclient:/$HOME/.kube/config
Back to kubclient change ownership of config file
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Now download calico manifest
curl https://docs.projectcalico.org/manifests/calico.yaml -O
Apply the manifest using following command
kubectl apply -f calico.yaml
Check all pod and wait until all pod is running
kubectl get pods --all-namespaces
Your HighAvailability cluster now ready to use.
Its important on production grade enterprise implementation to build kubernetes cluster with High Availability infrastructure so no single point of failure.
Cheers