Adding a New Node to a Cluster using Kubeadm

In this post we are going to use kubeadm to add a newly created Node to an existing cluster. The Lab consists of a cluster of Virtual Machines running on a Host under Virtualbox which has been built using Vagrant.

This means that each Virtual Machine has two NICs, the first one is used by Vagrant for management and the second is used for inter-host connectivity upon which the CNI overlay is created.

The use of the two NICs within the Virtual Machines means there are several things that need to be done to allow the cluster to be built in the first place using kubeadm, mostly related to ensuring that the api-server connection is via the correct interface.

By default the first interface that is created and used by Vagrant also acts as the default gateway, which is fine as it allows ssh connection from the Host and serves as a NAT interface for wider connection to the internet.

Looking at the IP address of the master node, after Docker and all the necessary Kubernetes components have been installed shows that the enp0s8 interface must be used for the inter-node communications.

We can also see the default route is set to the enp0s3 interface

By default when building a cluster using kubeadm the connections being made for the cluster will be via the default gateway which in our case is not correct. It is therefore necessary to explicitly set the correct IP address that will be used for the api server.

These settings are done when bringing up the master by setting --apiserver-advertise-address using the enp0s8 interface and --control-plane-endpoint using the DNS entry for the master.

kubeadm init --apiserver-advertise-address 192.168.200.20  --control-plane-endpoint k8s-master-1 --upload-certs | tee kubeadm-init.out

The kubeadm init has also been piped to a file to allow any fault finding to be done if there are any issues.

Once the kubeadm init command has been run the relevant settings can be seen below.

The first thing to do is to create the necessary kubeconfig file in a local user home directory that allows the running of kubectl. This is done by copying the necessary files as detailed in the kubeadm output.

Once this is done we can inspect the Pods that have been created and see that the CoreDNS Pods are pending and we can also see that the master Node is not ready.

These issues can be resolved by installing a suitable CNI network, which we will be do by installing weavenet.

kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Once this is done the CoreDNS Pods will become active and the suitable weavenet Pods will also be created (along with the other necessary objects that weavenet requires).

The worker Nodes can now join the cluster using the token that was generated by kubeadm. This is done by connecting to the worker Nodes and adding a static route to ensure it uses the correct interface to connect to the kubernetes cluster IP address and running the necessary join command.

The api server is run as a service that must be reachable by all the Nodes within the cluster. To confirm the IP address of all the services running the following kubectl command can be run on the master.

kubectl get service --all-namespaces

This confirms the IP address that needs to be reachable from the worker Nodes.

sudo ip route add 10.96.0.1/32 dev enp0s8

The static route is very important as without it the worker Node will send it's traffic via the default route (which is out of enp0s3) meaning it will never join the cluster.

Once this is done the first worker Node can be confirmed on the master.

The same thing is done for the second Node to allow it to join the cluster.

The initial token that is used to join the Nodes is valid for 24 hours which means that after this another token must be created to allow another Node to join the cluster. This could be due to maintenance or the expansion of the cluster.

Generate a New Token

We can easily view the list of tokens on the master by running

kubeadm token list

We must first generate a token and then print the join command associated with the token.

kubeadm token generate
kubeadm token create 1uth3n.fkhct68lnscgdqsr --print-join-command

The generated command can then be run on the worker Node in the normal way.

sudo kubeadm join k8s-master-1:6443 --token 1uth3n.fkhct68lnscgdqsr     --discovery-token-ca-cert-hash sha256:ed021289ab0fa3cbec56499fc87567d524eeffdcd36b73c5d832f0192664e776

Once this is done the new token and the number of Nodes on the master can be confirmed

kubeadm token list
kubectl get nodes

It can be seen that there is another token that has been created and the extra Node has joined the cluster.

Conclusion

Kubeadm is a tool that allows the easy creation of a Kubernetes cluster and will automatically generate all the necessary certificates and keys that are used to bootstrap the cluster.

When the command is first run it will automatically generate the necessary token that can be run on the worker Nodes that will allow them to join the cluster.

The token that is first generated will last for 24 hours before it expires. If there is a need to generate further tokens then kubeadm can be run again and can be used to generate the necessary join command that is run on a newly built Node.

When using multiple interfaces on the Hosts, which is done automatically if using Vagrant then care must be taken to ensure the inter-node communication is via the correct interface. If this is not done then the Nodes may not join the cluster.

It is also necessary to install a suitable CNI to allow the communication between within the cluster to work. If this is not done then the Nodes will not come up and the CoreDNS Pods will remain pending on the master.