My Blog

Node Selection in Kubernetes

Jason Salter -

The normal Kubernetes scheduler will select a suitable Node within the cluster but it is possible to manually schedule PODs on particular Nodes. This is useful if a particular application needs to be run on particular hardware or there are reasons that it should be deployed on particular servers.

In this post we have a simple cluster that has two worker nodes k8s-node-1 and k8s-node-2 and we'll use various ways of manually ensuring that our PODs are scheduled on a particular node.

NodeName

The simplest way of manually scheduling a POD is using the NodeName. This is part of the template spec within a Deployment and makes a simple link to the Node name.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nodeName-nginx
  name: nodename-nginx
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nodeName-nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodeName-nginx
    spec:
      nodeName: k8s-node-2  # This Line will select node
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: indexpage
          mountPath: /usr/share/nginx/html
      volumes:
        - name: indexpage
          configMap:
            name: nodename

This is another simple NGINX deployment which has a configMap mounted to provide a suitable index page.

vagrant@k8s-master:~/MyProject$ kubectl describe configmaps nodename --namespace=test
Name:         nodename
Namespace:    test
Labels:       <none>
Annotations:  
Data
====
index.html:
----
<h1>This is nginx-1</h1>
<p>This has been put on a node selected by the Nodename</p>

Events:  <none>

We'll expose the Deployment with a NodePort.

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nodeName-nginx
  name: nodename-nginx
  namespace: test
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nodeName-nginx
  type: NodePort

In a normal Deployment the PODs will be distributed equally over all Nodes (in our cluster's case all Nodes have been built in the same way, with the same CPU and memory capacity).

As we have used the nodeName in the template all the PODs are deployed on the same Node.

kubectl get pods --namespace=test -o wide

NodeSelector

With a NodeSelector we can assign our own labels to the Nodes within a cluster and use these to select where to run a POD.

To begin with we'll look at how our cluster Nodes are labelled by default.

kubectl get nodes --show-labels

The most significant label is one that is assigned automatically to the master Node. In normal operation working PODs are not run on the master Nodes and this is done by the node-role.kubernetes.io/master= label.

For our purpose we will not deploy our working PODs on the master but will add a label onto each of the worker Nodes.

kubectl label nodes k8s-node-1 colour=red
kubectl label nodes k8s-node-2 colour=blue

kubectl get nodes --show-labels

We now have our labels on the Nodes and will deploy some more PODs but have them just run on the Node that has a Red label. In this case this will just be a single Node but in a larger cluster this could be a number of Nodes we would be happy for the PODs to be run on.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nodeselector-nginx
  name: nodeselector-nginx
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nodeselector-nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodeselector-nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: indexpage
          mountPath: /usr/share/nginx/html
      nodeSelector:   ## This is the nodeSelector
        colour: red   ## This is the label for the Node
      volumes:
        - name: indexpage
          configMap:
            name: nodeselector

The Deployment manifest is slightly different as we are no longer relying on the Node name but a nodeSelector which has been set to the value of colour:red

We can then confirm our PODs are running on the desired Node in the normal way

kubectl get pods -o wide --namespace=test

We have confirmed that our second deployment has all 3 replicas running on the correct Node (k8s-node-1, which we had previously labelled colour=red). Of course we could now spray paint a physical server red but this would probably be taking things a little far.

This ability to link the deployments to labels that can be easily changed on Nodes does allow a lot of flexibility on where applications are run within a cluster. In a physical on-premises cluster this may allow easier maintenance as applications are moved from hardware that can then be modified or worked on.

NodeAffinity

Our last method is again going to make use of the labels that we have put on the Nodes. NodeAffinity is similar to the nodeSelector method but allows slightly more nuance and the ability to have a soft/preference option rather then a hard selector.

The options that can be used are also more expressive then a simple yes/no operator.

We'll set up another deployment of 3 NGINX replicas which will run on our Node with a colour:blue (k8s-node-2).

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nodeaffinity-nginx
  name: nodeaffinity-nginx
  namespace: test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nodeaffinity-nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nodeaffinity-nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: colour
                operator: In
                values:
                - blue
      containers:
      - image: nginx
        name: nginx
        volumeMounts:
        - name: indexpage
          mountPath: /usr/share/nginx/html
      volumes:
        - name: indexpage
          configMap:
            name: nodeaffinity

It can be seen that there is quite a large affinity section within the spec of the template. This section would be used for all affinity rules being implemented, which can also include affinity rules linked to the placement of other PODs within the cluster as well as Nodes. This can also include anti-affinity rules as well, meaning that the placement of PODs could be kept away from certain Nodes or PODs.

In this example we are keeping things simple by matching the values colour:blue within a Node label.

Once we run the Deployment we can confirm that the new PODs have been put onto the correct Node.

kubectl get pods -o wide --namespace=test

We can confirm that the nodeaffinity PODs have all ended up on the same Node which has the colour:blue label.

Conclusions

This post has run through the basic ways that the kubernetes scheduler can be over-ridden to allow applications to be deployed on particular Nodes within the cluster.

The simplest, but most inflexible way, is to make use of the nodeName within the POD spec. This is simple to do but does mean that it is linked directly to the Node's name which is not easily changed once it has been deployed.

The second method makes use of the user-defined labels that can be easily added and removed from Nodes and uses the nodeSelector option. This is another easy option and is probably a better method then the cruder nodeName method.

The final method makes use of affinitys, of which nodeAffinity is just one option. This gives most flexibility and can be used to construct quite complex rules on where to run PODs (something we didn't make use of in this post).

The manual over-riding of the scheduler should be carefully thought about as it is very possible to get into trouble with unexpected results. It however is a powerful tool that can be made use of to maximise resources and performance.

The use of labels to influence behaviour is generally much more preferred should there be a requirement to manipulate the placement of PODs within the cluster.