Ever since I started using Kubernetes I have always been on the pursuit of finding the right tools to monitor. and I did find a lot of products from the Open source community and here are my favourites
Despite having all these toolsets, I was looking for a way to find out the right tool that gives me an eagle-eye view of the node usage and makes me able to see the node level, Namespace level and pod level CPU and Memory utilization
kubectl top pods did not impress me much and I had a lot of wrappers around this to meet my need. but it was not enough
But the one that gained my confidence and proved to solve my requirement is kubectl resource-capacity
It is not an inbuilt kubectl command. we are going to add this extra plugin using krew it is a package manager for Kubectl, something like npm for Node
In this article, we are going to see how to efficiently get the node and pods CPU and memory usage in our Kubernetes cluster with nice formatting and sorting.
It works well with all the Kubernetes clusters.
- Elastic Kubernetes Service ( AWS)
- Google Kubernetes Engine (GKE)
- Azure Kubernetes Service (AKS)
Introducing our Open Source tool - Kube Node Usage
I have created a new open-source tool named Kube-Node-Usage to list the nodes with their CPU, Memory and Disc usage
Here is a quick screenshot of the tool
Like what you just saw? Go ahead and download and try it
it is available on the following GitHub repository
https://github.com/AKSarav/Kube-Node-Usage
We have released new features in Version 2 of Kube Node Usage
- Sort By Usage, Node, Free, and Max values of CPU, Memory and Disk Usage
- Select a Specific Node using the Filter
- Filter only by the colour of the usage ( Red, Green, Yellow)
- Continuously monitor the usage in a specific interval on specific colour or node name
- Now the results are directly fetched through kubectl top for more reliable/real-time output
Try it out and let me know your feedback.
Now to another solution. with Krew and the resource capacity plugin. but let's start with a prerequisite - Package Manager for Kubectl called Krew
What is Krew and how to download
Krew is a plugin manager for kubectl command-line utility
From Krew's website:
Krew helps you:
- discover kubectl plugins,
- install them on your machine,
- and keep the installed plugins up-to-date.
There are 174 kubectl plugins currently distributed on Krew.
Krew works across all major platforms, like macOS, Linux and Windows.
you can find the download and installation instructions of Krew here
Once you have installed the krew validate if the installation is successful by issuing the kubectl krew version command
⚡ ⇒ kubectl krew version OPTION VALUE GitTag v0.4.2 GitCommit 6fcdb79 IndexURI https://github.com/kubernetes-sigs/krew-index.git BasePath /Users/saravananthangaraj/.krew IndexPath /Users/saravananthangaraj/.krew/index/default InstallPath /Users/saravananthangaraj/.krew/store BinPath /Users/saravananthangaraj/.krew/bin DetectedPlatform darwin/amd64
Install resource-capacity plugin using Krew and Kubectl
We presume that your krew installation is successful with that done. You can install 174+ amazing plugins
In our case, we are going to install resource-capacity plugin
kubectl krew install resource-capacity
Once the plugin is installed. you can validate if the plugin is installed successfully by validating the list of plugins installed and their version
⚡ ⇒ kubectl krew list PLUGIN VERSION ctx v0.9.4 krew v0.4.2 ns v0.9.4 resource-capacity v0.6.1
In our case, the version of resource-capacity is 0.6.1
Now we will use the various commands available with resource-capacity and analyze the results.
Get Node CPU usage and memory usage of each node - Kubectl
The Simple resource-capacity command with kubectl would return the CPU requests and limits and memory requests and limits of each Node available in the cluster.
kubectl resource-capacity
You can use the ‐‐sort cpu.limit flag to sort by the CPU limit. there are more sorts available we will see next.
kubectl resource-capacity ‐‐sort cpu.limit
By default, this would not return the Utilization data for each node.
If you need to utilization report for each node you need to use ‐‐util flag
Node CPU usage and memory usage with More Sorting
As we have seen above kubectl resource-capacity supports sort by fields.
the sort flag supports the following values
- cpu.util
- cpu.request
- cpu.limit
- mem.util
- mem.request
- mem.limit
- name
if not applied, by default it is sorted in ascending order by the name
# CPU Usage with Sort options kubectl resource-capacity ‐‐sort cpu.limit kubectl resource-capacity ‐‐sort cpu.request kubectl resource-capacity ‐‐sort cpu.util # Memory Usage with Sort options kubectl resource-capacity ‐‐sort mem.util kubectl resource-capacity ‐‐sort mem.request kubectl resource-capacity ‐‐sort mem.limit
Including Pod level usage with Node Usage report
In the previous commands, we have seen, if you add a new flag ‐‐pods it would show the pod level usage for each node
kubectl resource-capacity ‐‐sort cpu.util ‐‐util ‐‐pods
The output of this command would give a result like this. it would contain the Node CPU and Memory statistics along with the POD level usage as well.
You can change the ‐‐sort to your choice. in my case it is sorted based on CPU
Including Container and Pod level usage with Node Usage report
In the previous command, we have seen, using ‐‐pods would add statistics of the POD level CPU and memory usage
We can further break it down to finer detail such as containers in each pods by adding ‐‐containers flag
kubectl resource-capacity ‐‐sort cpu.util ‐‐util ‐‐pods ‐‐containers
You can see we have added ‐‐containers to our previous command
How to get the CPU and Memory Usage of the namespace
This feature is widely used by me. Whenever I want to find out which namespace is consuming more memory or CPU I use this command
Also, to Quickly find out which nodes are hosting the pods of the particular namespace and their capacity I use this command
As we have already seen we can use ‐‐pods or -p and ‐‐containers or -c to get the finer usage detail of Pods and Containers as well.
To find out the Usage of kube-system namespace I can use the following command
kubectl resource-capacity -n kube-system -p -c or kubectl resource-capacity -n kube-system ‐‐pods ‐‐containers
The output of this command would be something like this
You can see that the CPU and Memory usage of each Container and POD in the namespace is shown.
This would help us to do better resource capacity planning and rightsizing.
Using Labels to get CPU and Memory Usage - Kubectl
You can filter the nodes based on their label and get the CPU and memory usage of these specific nodes by using ‐‐node-labels flag
Here is the commands with node labels.
In our case, we want to filter the nodes with this label node.kubernetes.io/instance-type=t3a.large and get the resource capacity and usage of those nodes
kubectl resource-capacity ‐‐node-labels node.kubernetes.io/instance-type=t3a.large
Output of this command looks like this
You can filter nodes and resources based on three lebels they are listed below.
‐‐pod-labels- Pod Level Labels‐‐namespace-labels- Labels used at the Namespace Level‐‐node-labels- Labels used at the node level
Wait. the output is not just a table, there is more.
Kubectl resource-capacity is configured to give output as a table by default but the output can be customized into other formats too
The supported output formats are
- yaml
- json
- table ( default)
Since it is supporting multiple output options. It can fit right into your automation frameworks or tools.
You can define the output of the kubectl resource-capacity by using the flag -o or ‐‐output
Conclusion
In this article, we have discussed about how to install and configure krew plugin manager for kubectl and how to use resource-capacity plugin with kubectl to get resource usage at various levels
We learnt various things like
- How to list nodes based on the CPU or memory utilization or usage
- Get CPU and Memory usage of namespaces and their corresponding nodes
- Sort nodes and Pods based on the CPU and Memory consumption
- Doing CPU and Memory usage monitoring with Kubectl
- Listing containers and pods with their CPU and Memory Allocation and Consumption
Hope it helps in monitoring and right-sizing your Kubernetes Cluster.
Do you know any better CLI plugins or monitoring options to manage the CPU and Memory Usage on Kubernetes Cluster? Let us know in the comments
Cheers
Sarav AK
Follow me on Linkedin My Profile Follow DevopsJunction onFacebook orTwitter For more practical videos and tutorials. Subscribe to our channel
Signup for Exclusive "Subscriber-only" Content
