Kubectl Scale down Pods and Save Cost - Kubernetes | Devops Junction

As the industry is going heavy on Kubernetes, as more and more monoliths are converted to microservices and onboarded to Kubernetes, as the number of namespace and nodes increase in your Kubernetes cluster, the cost of running Kubernetes would also increase proportionally

There are a lot of cost optimization ideas for Kubernetes, one such idea is to hibernate or scale down your namespaces when they are in need.

Let's say, your organization is using Kubernetes to power the  DEV/QA/UAT and internal environments and you have no weekend work policy or reduced weekend work policy.

Or let's say you are in a specific timezone and you do not want these environments by the night and need it only by the day.

You can choose to hibernate or scale down these environments every night and every weekend to reduce the cost.

Of course, you need to leave some of the namespaces online such as kube-system and monitoring workspaces like grafana or prometheus or any business critical namespaces too.

Python Script to scale down all the pods with exceptions

Here is a python script to scale down all the workspaces with some exceptions

You can schedule this script via some automation or scheduler like Jenkins to run every Friday or every evening around 9 pm as per your needs

#!/bin/python

import os
import subprocess
import re
import sys


exclusions=sys.argv[1:]

print("Excluding these namespaces",exclusions)

namespaces = subprocess.run(["kubectl", "get", "namespaces", "-o", "name"],
                            capture_output="True",
                            timeout=10)
# print(namespaces)
if namespaces.stdout:
    for ns in namespaces.stdout.decode("utf-8").split("\n"):
        ns = ns.replace("namespace/", "")
        if ns in exclusions and re.match(r'^kube-', ns) is not None:
            continue

        pods = subprocess.run(["kubectl", "get", "pods", "-n", ns],
                              capture_output="true",
                              timeout=10)
        if pods.stderr and "No resources found" in pods.stderr.decode("utf-8"):
            print("\n – %s - Namespace Empty" % ns)

        else:
            if ns not in exclusions and re.match(r'^kube-', ns) is None:
                print("\n – %s - Scaling down" % ns)
                scale = subprocess.run([
                    "kubectl", "scale", "deployment", "--all", "--replicas=0",
                    "-n", ns
                ],
                                       timeout=100)
                if scale.returncode == 0:
                    print("✓ %s - Namespace Scaled Down" % ns)

 

Schedule the script with Jenkins

You can schedule this script with Jenkins.

For Jenkins to be able to connect to the cluster, you need to have the kubectl and the necessary permissions. We assume you have done that.

Once you have ensured that you can reach the Kubernetes cluster and execute the Kubectl command from the Jenkins machine.

you can do the following

1. Create a Free Style Job
2. Add Parameterized String input to accept the exclusions ( Instruct the user with a description to pass space-separated values)
3. Execute Shell as  Build Step and run the script with pre defined and user defined exclusions like this

 

$ python3 /var/lib/jenkins/scripts/scale-down-alfred.py "common" "default" "karpenter" "velero" "traefik-dev" "observability" $Exclusions

You can see we are passing the user-defined exclusions along with pre-defined (defined explicitly in the command) namespaces.

In case you want to add any namespace permanently you can update this command in the Jenkins job itself (or) pass it on-demand when Build with parameters

 

Refer to our other posts in Kubernetes here 

Cheers

Sarav

---