In Kubernetes, managing resource consumption is an important aspect of production. All the resources must be carefully verified before deploying them in production. 

‍

The Kubernetes pods immediately start using the required resources once they are scheduled. If the resource requirements are not defined for the pods then it can lead to out-of-memory issues and it can interfere with other pods running on the same nodes. Due to this misconfiguration, other pods can also face resource starvation and can go in pending state.

‍

The unused allocated resources can cause a financial burden and that money can be used for other useful projects. In order to maintain the health of a cluster it’s crucial to consider resource requirements at the beginning. In this article, we will explore how to manage the resource requirements in Kubernetes.

Managing Resource Requests & Limits

Let’s consider an example in which we have two containers running in a pod. Here the sidecar container is a facilitator for running the main container. 

<pre class="codeWrap"><code>apiVersion: v1
  kind: Pod
  metadata:
   name: frontend
  spec:
   containers:
   - name: app
     image: images.my-company.example/app:v4
     resources:
       requests:
         memory: "64Mi"
         cpu: "250m"
       limits:
         memory: "128Mi"
         cpu: "500m"
   - name: log-aggregator
     image: images.my-company.example/log-aggregator:v6
     resources:
       requests:
         memory: "64Mi"
         cpu: "250m"
       limits:
         memory: "128Mi"
         cpu: "500m"
</code></pre>

The Resource.requested and Resources.limited fields are used to track the resource request and limit information. The resource.requested is the sum of limited resources for all its containers and Resources.limited is the sum of limited resources for all of its containers.

‍

To know which node is best for scheduling pod fields are used. Here the Kubernetes control plane is used to detect the available resources on each container and compare it with incoming requests. After this, it automatically assigns the nodes to the available pods.

CPU Resource Requests and Limits

The CPU resource requests and limits are time-sensitive operations and are calculated in milliseconds. As an example, if a container requires one core then you need to add the value “1000m” similarly, or ¼ the value would be “250m”.

‍

This seems a fairly simple calculation but even the slightest mistake can crash the pods. For a quad-core machine if we request 5000m accidentally then pods can never be scheduled. 

‍

Even if the CPU hits the resource limit it doesn’t stop the container but can slow down the whole application.

Memory Resource Requests and Limits

The memory requests and limits are defined in bytes. Commonly used value for memory is mebibyte but other units like bytes or petabytes can also be used.

‍

Just like CPU resources, over-requested memory can also cause problems. If you request memory more than the available memory then pods can’t be scheduled. Unlike CPUs, if memory requests fall out of limit the pods are terminated.

‍

You can restart the pod using kubelet if it’s restartable. Kubernetes automatically restarts a pod that exceeds the memory limits. But if this behavior continues, pods can be evicted. The eviction of pods means that pods are forcefully terminated. Kubelets kill the containers and set the status of pods as failed till the memory usage is back to normal.

Requests & Limits Using Namespaces

Using Kubernetes namespaces, clusters are divided into multiple virtual clusters. These virtual clusters can be then assigned to specific services or teams. This approach is useful when multiple teams are working within the same Kubernetes cluster.

‍

You should also define the resource requirements for each namespace in order to make sure that no resources are reserved without being used. This can be done by assigning resource quotas and limit ranges to namespaces.

Resource Quotas

In a specific namesake resource quotas are the number of compute resources allocated to a virtual cluster. These resources can only be used by that namespace. This can make sure that the infrastructure cost does not exceed the defined limit. Resource quotas can be used to limit namespaces to create a specific number and type of resources.

‍

To understand this let’s create a resource quota on two pods. This means that you can only create 2 pods in that specific namespace. Run the following command to create resource quota names pods-test.

<pre class="codeWrap"><code>cat &lt&ltEOF | kubectl apply -f -
 apiVersion: v1
 kind: ResourceQuota
 metadata:
  name: pod-examples
 spec:
  hard:
    pods: "2"
 EOF</code></pre>

‍

After the resource quota is created, create three different pods by running the following command three times.

<pre class="codeWrap"><code>kubectl run pod1 --image=nginx</code></pre>

The third command will fail as our resource quota has reached its limit.

‍

Resource quotas within namespaces help to avoid exceeding resource limits. However, it is possible for one object to use all the resources and creates resource starvation for the rest of the resources. In these scenarios, limit ranges are helpful as they help to set minimum and maximum limits on CPU, RAM, and storage requests per PVC within a namespace.

‍

Let’s take an example and define 1 GB as the maximum limit and 500 MB as the minimum limit of memory allocation. Use the following command to set these limits in the default namespace.

<pre class="codeWrap"><code>cat &lt&ltEOF | kubectl apply -f -
 apiVersion: v1
 kind: LimitRange
 metadata:
  name: min-max-memory-demo
 spec:
  limits:
  - max:
      memory: 1Gi
    min:
      memory: 500Mi
    type: Container
 EOF</code></pre>

‍

To verify that the limit range is working we will create a container with a resource limit of more than 1 GB and memory requirements below 500 MB

As our specified criteria fall out of the limit, the pod creation will fail and give the following error:

<pre class="codeWrap"><code>Error from server (Forbidden): error when creating "STDIN": pods "memory-demo-pod" is forbidden: [minimum memory usage per Container is 500Mi, but request is 100Mi, maximum memory usage per Container is 1Gi, but limit is 1536Mi]</code></pre>

Challenges with Resource Requests and Limits

A common challenge in managing Kubernetes resources is resource reservation issues like under and overcommitments. There are settings like namespaces or resource limits available but still clusters can face such issues and resources can be wasted. In the production environment, the testing pods can lead to application outages. If the allocated resources are less than the requirement then it can lead to out-of-memory issues and applications can crash.

‍

The possible solution to resolve this issue is to use pod priority and preemption. Using this approach if a higher priority pod is in queue the lower priority pod gets evicted. Let’s take an example and create a PriorityClass to implement pod priorities.

<pre class="codeWrap"><code>apiVersion: v1
 kind: PriorityClass
 metadata:
   name: critical
 value: 9999999
 globalDefault: false
 description: "Use this class for critical service pods only."</code></pre>

‍

Disable the  preemption of a high PriorityClass by setting the preemptionPolicy to “Never.”

<pre class="codeWrap"><code>apiVersion: v1
 kind: PriorityClass
 metadata:
   name: high-priority
 value: 7777777
 globalDefault: false
 preemptionPolicy: Never
 description: "Use this class for critical service pods only."</code></pre>

‍

You can see that using the priorityCassName field pod priorities are mapped back to a pod.

<pre class="codeWrap"><code>apiVersion: v1
 kind: Pod
 metadata:
   name: a-pod
 spec:
   containers:
   - name: a1-container
     image: nginx
     imagePullPolicy: IfNotPresent
     resources:
         limits:
          memory: “1Gi”
          cpu: “500Mi”
  - name: a2-container
     image: nginx
     imagePullPolicy: IfNotPresent
     resources:
         limits:
          memory: “1Gi”
          cpu: “500Mi”
   priorityClassName: critical</code></pre>

This way, pod priority and preemption can be used to deal with over-provisioning of resources in a cluster.

‍

A Kubernetes cluster can have hundreds of pods and resources running at the same time. This makes it difficult to create a utilization report for individual applications on a shared cluster. You can use the labels which are simple key/value pairs and assign it to objects like pods and namespaces.

‍

This will allow you to represent the cost centers. This will help you to logically organize the objects and perform operations on subsets of objects. Consider the following examples for pod and container labels.

<pre class="codeWrap"><code>apiVersion: v1
 kind: Pod
 metadata:
   name: a-pod
   labels:
        env: staging
     team: infra
  spec:
   containers:
   - name: a1-container
     image: nginx
     imagePullPolicy: IfNotPresent
     resources:
         limits:
          memory: “1Gi”
          cpu: “500Mi”
  - name: a2-container
     image: nginx
     imagePullPolicy: IfNotPresent
     resources:
         limits:
          memory: “1Gi”
          cpu: “500Mi”
   priorityClassName: critical</code></pre>

Annotations can also be used on containers but annotations don’t allow you to select and identify objects. Aligning applications against cluster resources can be used. For instance, if your cluster is deployed using kOps then CloudLabels and NodeLabels both are supported. Labels that are applied to kOps can be applied to other instances in the same cluster. These act as tags and can be utilized in cost allocation and chargeback.

‍