The standard output options available with the kubectl command allow gathering of information about our Kubernetes cluster and what is running within it. In many circumstances this may be sufficient but sometimes there is a need to display information not part of the standard kubectl output.
For example it is relatively easy to determine what Pods are running in a particular namespace.
kubectl get pods
The output tells us which Pods are running within the namespace and we can also get a bit more information by adding the -o wide flag.
kubectl get pods -o wide
This gives us more information, including the actual Node the Pod is currently running on and it's IP address. However there is no indication of which image has been used or what container ports are in use.
To get this extra information it is necessary to describe the Pod or Deployment which returns a lot of additional information which may or may not be of interest.
kubectl describe pod website1-7758c6fd5-db5sf
Name: website1-7758c6fd5-db5sf
Namespace: test
Priority: 0
Node: k8s-node-1/10.53.104.77
Start Time: Sun, 31 Jan 2021 16:29:05 +0000
Labels: app=website1
pod-template-hash=7758c6fd5
project=blog-entry
Annotations: <none>
Status: Running
IP: 10.44.0.1
IPs:
IP: 10.44.0.1
Controlled By: ReplicaSet/website1-7758c6fd5
Containers:
nginx:
Container ID: docker://a91b2c9de7596afca05a68c0cdf547852cb40241c8a799520f44ce7f89dc969c
Image: nginx:1.18.0-alpine
Image ID: docker-pullable://nginx@sha256:7ae8e5c3080f6012f8dc719e2308e60e015fcfa281c3b12bf95614bd8b6911d6
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 31 Jan 2021 16:29:10 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-jwtw5 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-jwtw5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-jwtw5
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36m default-scheduler Successfully assigned test/website1-7758c6fd5-db5sf to k8s-node-1
Normal Pulling 36m kubelet Pulling image "nginx:1.18.0-alpine"
Normal Pulled 36m kubelet Successfully pulled image "nginx:1.18.0-alpine" in 4.286046676s
Normal Created 36m kubelet Created container nginx
Normal Started 36m kubelet Started container nginx
kubectl describe deployment website1
Name: website1
Namespace: test
CreationTimestamp: Sun, 31 Jan 2021 16:29:05 +0000
Labels: app=website1
project=blog-entry
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=website1
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=website1
project=blog-entry
Containers:
nginx:
Image: nginx:1.18.0-alpine
Port: 8080/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: website1-7758c6fd5 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 37m deployment-controller Scaled up replica set website1-7758c6fd5 to 1
As can be seen describing the Pods and Deployments generates a lot of additional information which is limited to those particular objects. In our case we would like to compare this information across all Pods running in the namespace, which would mean running the command for each Pod or Deployment.
When kubectl is used it actually interrogates the api-server and pulls back the complete json data for the whole object, although by default this is displayed in a cut-down form.
It is also possible to output in either yaml or json format which is often too much information.
kubectl get pods -o yaml
kubectl get pods -o json
However it is worth taking the time to become familiar with the full json data being returned by the kubectl commands and learn how to use jsonpath queries to extract desired information.
Using output custom columns allows information extracted by these jsonpath queries to be displayed in the familiar format used by standard kubectl commands.
In this example we will extract the image and container ports from the returned status of the Pods.
We will then display these details along with the Pod name, Node details and Pod IP details. This will give a single summary report allowing us to view the information for all Pods in the namespace at the same time.
kubectl get pods -o custom-columns=NAME:.metadata.name,IP:.status.podIPs[*].ip,PORT:.spec.containers[*].ports[*].containerPort,IMAGE:.status.containerStatuses[*].image,NODE:.spec.nodeName
The use of custom columns has enabled us to get a formatted list that looks similar to the original output but also including the image and container port details.
We can now see that my-test-5dfcd48dd9-dc88v is actually a busybox Pod, which would not be obvious from it's name. We can also see that the nginx Pods are exposing port 8080 which must match any associated services.
We can also see the two Pods running nginx are not running the same version of nginx (which may or may not be an issue)
In this initial case the command requires a lot of typing but once a query has been checked it is possible to save it to a template file which allows it to be used in the future.
The format of the file is similar to the typed information.
cat my-report.txt
To run the report it is a simple command to reference the file while setting the output format to custom columns.
kubectl get pods -o custom-columns-file=my-report.txt
Conclusions
When there is a need to generate reports needing information normally only available by describing an object or trawling through the full .yaml or .json outputs the use of custom columns is very useful. The custom columns reports can give a good way of making comparisons of objects such as Pods and their current status.
While this still means the construction of suitable jsonpath query expressions once these are done and the report is in the correct format it is possible to write a template form that makes it easy to run the query again in the future. These template files can be used on any machine that has access to the api-server, making them available for scripting and automation in the future.
It is very much worth taking the time to study the json format of returned api-calls to get an idea of what information is available, as this can save time in fault-finding.
In this particular example we now have a simple way of looking at the images that have been used for the running Pods as well as their containerports.