Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Kubernetes Resource Quotas

Kubernetes Resource Quotas

This article shows you how to define quotas on Kubernetes namespaces to allow for the sharing of computing resources.

By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

When several teams of people share Kubernetes development and production nodes (servers running Kubernetes), it is typically a requirement to divide the computing resources equally (CPU, RAM, various types of disk space).

Kubernetes namespaces help with this via creating logically isolated work environments. But namespaces does not enforce limitations / quotas.

We need to use Kubernetes quotas to precisely specify strict quota limits for around 15 Kubernetes API resources.

All the exercises in this tutorial run in the default namespace. At your work you need to divide your nodes into namespaces and define quotas for each of those ... separately for development and for production. That specific topic is outside the scope of this tutorial.

This tutorial focus on how to define quotas.

Defining quotas are easy. Checking to see quotas get enforced is easy too, so this tutorial is quite repetitive. Just reading the Kubernetes docs does not let the facts sink in, you need practice and experience, hence this tutorial.

1) Basic Quota : 2 Pods

Enter this using your favorite Linux editor.

It defines a ResourceQuota that specifies a hard limit of 2 Pods.

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: count-quotas
spec:
  hard:
    pods: "2"

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Get a list of all quotas on your node.

kubectl get quota

NAME            CREATED AT
object-counts   2019-01-22T05:53:02Z

Currently only 1 quota object: the one just created.

Describe how Kubernetes interpreted our YAML spec file.

kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Resource    Used  Hard
--------    ----  ----
pods        0     2

Quota of 2 Pods available, zero used so far.

Let's create one Pod to use one of the available Pods.

nano myQuota-Pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-1
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']
         
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Nothing special about this Pod spec. Pods need no special action or spec definition to use quota-limited resource.

Quotas are automatically enforced.

WARNING. All Pod specs in this tutorial specifies terminationGracePeriodSeconds: 0

( By default, all deletes are graceful within 30 seconds. This allows time for shutdown routines to run. )

I do this to speed of the deletion of the many Pods throughout this tutorial. Only adjust terminationGracePeriodSeconds during production after careful consideration.

Create the Pod.

kubectl create -f myQuota-Pod.yaml

pod/quota-pod-1 created

We want to exceed quota, so we need more Pods.

Edit just the name of the Pod in myQuota-Pod.yaml

  name: quota-pod-2

Create this second Pod.

kubectl create -f myQuota-Pod.yaml

pod/quota-pod-2 created

Investigate the status of our quota :

kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Resource    Used  Hard
--------    ----  ----
pods        2     2

2 Pods created so far, full quota used up.

Here are our Pods:

kubectl get po

NAME          READY   STATUS    RESTARTS   AGE
quota-pod-1   1/1     Running   0          2m36s
quota-pod-2   1/1     Running   0          90s

We want to exceed quota, so we need 1 more Pod.

Edit just the name of the Pod in myQuota-Pod.yaml

  name: quota-pod-3

Create the Pod.

kubectl create -f myQuota-Pod.yaml

Error from server (Forbidden): error when creating "myQuota-Pod.yaml": pods "quota-pod-3" is forbidden: exceeded quota: object-counts, requested: pods=1, used: pods=2, limited: pods=2

As expected, create failed ( Quota enforced ). Error message easy to interpret.

This ends this demo:

  • create a quota to limit usage of a resource
  • usage within quota limit allowed
  • attempt to exceed quota gets a well-deserved error message

Delete Pods and resourcequota objects.

kubectl delete pod/quota-pod-1
pod "quota-pod-1" deleted

kubectl delete pod/quota-pod-2
pod "quota-pod-2" deleted

kubectl delete quota/object-counts
resourcequota "object-counts" deleted

kubectl get po
No resources found.

2) CPU Quota on Requests and Limits

CPU resources are usually high on list of resources to limit via quotas.

Kubernetes supports 2 CPU quotas:

  • requests ... a Pod requests an amount of CPU resources
  • limit ... a Pod defines the limit of CPU resources it will use

Below is our ResourceQuota for :

  • all Pods in total may request 1 full CPU = 1000 Millicores
  • all Pods in total may limit 2 full CPUs = 2000 Millicores

( One Millicores is 1/1000 of a CPU, therefore 1000m equals 1 CPU. 1000m equals one CPU on all computers. )

A four core server has a CPU capacity of 4000m.

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    requests.cpu: "1000m"
    limits.cpu: "2000m"

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Now we define a Pod that will use a part of those resources.

nano myQuota-Pod-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-1
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']

    resources:
      requests:
        cpu: "500m"
      limits:
        cpu: "1500m"
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Define a Pod that will use the remaining part of those resources.

nano myQuota-Pod-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-2
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']

    resources:
      requests:
        cpu: "500m"
      limits:
        cpu: "500m"
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Define a Pod that will exceed the CPU resources quota. ( When running simultaneously with 2 previous Pods )

nano myQuota-Pod-3.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-3
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']

    resources:
      requests:
        cpu: "10m"
      limits:
        cpu: "50m"
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Create the first 2 Pods.

kubectl create -f myQuota-Pod-1.yaml
pod/quota-pod-1 created

kubectl create -f myQuota-Pod-2.yaml
pod/quota-pod-2 created

Investigate if quota usage is as expected.

kubectl describe quota object-counts

Name:         object-counts
Namespace:    default
Resource      Used  Hard
--------      ----  ----
limits.cpu    2     2
requests.cpu  1     1

Our 2 Pods use exactly the total CPU quotas.

Attempt to use even more CPU resources.

kubectl create -f myQuota-Pod-3.yaml

Error from server (Forbidden): error when creating "myQuota-Pod-3.yaml": pods "quota-pod-3" is forbidden: exceeded quota: object-counts, requested: limits.cpu=50m,requests.cpu=10m, used: limits.cpu=2,requests.cpu=1, limited: limits.cpu=2,requests.cpu=1

As expected: even exceeding quota by 10% results in error.

Demo done, delete ...

kubectl delete -f myQuota-Pod-1.yaml
pod "quota-pod-1" deleted

kubectl delete -f myQuota-Pod-2.yaml
pod "quota-pod-2" deleted

kubectl delete quota/object-counts
resourcequota "object-counts" deleted

3) Quota Count/ Syntax

We will now create a quota of 2 secret API objects using the previous syntax.

Then we test its enforcement.

Then we define the quota using count/ syntax.

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    secrets: "2"

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Describe quota:

kubectl describe quota object-counts
Name:       object-counts
Namespace:  default
Resource    Used  Hard
--------    ----  ----
secrets     1     2

Unexpectedly one secret already used ?

List all current secrets:

kubectl get secrets

NAME                  TYPE                                  DATA   AGE
default-token-gs2wt   kubernetes.io/service-account-token   3      26d

This is the default secret token automatically created by Kubernetes. It is used by Pods to access the Kubernetes API.

So this secret uses 1 of our 2 quota slots.

One very simple and INSECURE way to create a secret is to use this one-liner command:

kubectl create secret generic my-insecure-secret-1 --from-literal=literalkey1=insecure-secret-value-1
secret/my-insecure-secret-1 created

Run it to create a second secret.

List all secrets:

kubectl get secrets

NAME                   TYPE                                  DATA   AGE
default-token-gs2wt    kubernetes.io/service-account-token   3      26d
my-insecure-secret-1   Opaque                                1      3s

Describe the quota status:

kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Resource    Used  Hard
--------    ----  ----
secrets     2     2

As expected, 2 secrets using full quota.

Attempting to create another secret results in error.

kubectl create secret generic my-insecure-secret-2 --from-literal=literalkey2=insecure-secret-value-2
Error from server (Forbidden): secrets "my-insecure-secret-2" is forbidden: exceeded quota: object-counts, requested: secrets=1, used: secrets=2, limited: secrets=2

This used the ( by now familiar ) first syntax.

Delete my-insecure-secret-1:

kubectl delete secret/my-insecure-secret-1
secret "my-insecure-secret-1" deleted

Delete quota object:

kubectl delete quota/object-counts

Second syntax for specifying count limit quotas:

Just prefix the API object you want to quota with count/ 

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    count/secrets: "2"

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Describe the quota.

kubectl describe quota object-counts

Name:          object-counts
Namespace:     default
Resource       Used  Hard
--------       ----  ----
count/secrets  1     2

It works exactly as the previous syntax quota.

It correctly already counts the default secret token as one secret already being used.

kubectl get secrets

NAME                  TYPE                                  DATA   AGE
default-token-gs2wt   kubernetes.io/service-account-token   3      26d

The effect of syntax 1 and 2 is identical:

Syntax 1:

spec:
  hard:
    pods: "2"
    secrets: "2"

Syntax 2:

spec:
  hard:
    count/pods: "2"
    count/secrets: "2"

Pick one of these as your standard way of defining quotas.

Delete Quota

kubectl delete -f myQuota.yaml

resourcequota "counts-quota" deleted

From https://kubernetes.io/docs/concepts/policy/resource-quotas/#object-count-quota

List of resources you can limit via object count quota:

  • count/persistentvolumeclaims
  • count/services
  • count/secrets
  • count/configmaps
  • count/replicationcontrollers
  • count/deployments.apps
  • count/replicasets.apps
  • count/statefulsets.apps
  • count/jobs.batch
  • count/cronjobs.batch
  • count/deployments.extensions

4) Quota on Quality of Service (QoS) Class

You can limit Pods based on their Quality of Service (QoS) class.

From https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-besteffort

For a Pod to be given a QoS class of BestEffort , the Containers in the Pod must not have any memory or CPU limits or requests.

Let's define a quota of 2 Pods in the BestEffort QoS class.

Terminology: Pods running in the BestEffort class are in the scope of the quota.

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    count/pods: "2"
  scopes:
  - BestEffort

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Describe the quota:

kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource    Used  Hard
--------    ----  ----
count/pods  0     2

It clearly describes the BestEffort scope.

Now we need to define BestEffort Pods: one that does not have resource requirements.

None of the Pods below have such lines in their YAML spec files.

    resources:
      requests:
        cpu: "500m"
      limits:
        cpu: "500m"

We need 3 Pods since 2 Pods will fall within quota limits, the 3rd will attempt to exceed quota.

( Only Pod names differ below )

nano myQuota-Pod-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-1
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
nano myQuota-Pod-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-2
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0
nano myQuota-Pod-3.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-3
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Create the first Pod.

kubectl create -f myQuota-Pod-1.yaml
pod/quota-pod-1 created

Check quota stats are as expected.

kubectl describe quota object-counts
Name:       object-counts
Namespace:  default
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource    Used  Hard
--------    ----  ----
count/pods  1     2

Our Pod correctly classified as BestEffort. It counts as 1 Pod against the quota.

Create 2nd Pod.

kubectl create -f myQuota-Pod-2.yaml
pod/quota-pod-2 created

Check quota again.

kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Scopes:     BestEffort
 * Matches all pods that do not have resource requirements set. These pods have a best effort quality of service.
Resource    Used  Hard
--------    ----  ----
count/pods  2     2

As expected: bot Pod quotas used.

Attempt to create a third BestEffort Pod:

kubectl create -f myQuota-Pod-3.yaml
Error from server (Forbidden): error when creating "myQuota-Pod-3.yaml": pods "quota-pod-3" is forbidden: exceeded quota: object-counts, requested: count/pods=1, used: count/pods=2, limited: count/pods=2

Fails since it exceeds quota.

If we kubectl describe our first 2 Pods we will see their QoS is BestEffort.

kubectl describe pod/quota-pod-1|grep Best
QoS Class:       BestEffort

kubectl describe pod/quota-pod-2|grep Best
QoS Class:       BestEffort

Done, delete ...

kubectl delete -f myQuota-Pod-1.yaml
pod "quota-pod-1" deleted

kubectl delete -f myQuota-Pod-2.yaml
pod "quota-pod-2" deleted

kubectl delete quota/object-counts
resourcequota "object-counts" deleted

5) Scopes: NotTerminating Quotas

Another scope is NotTerminating

Matches all pods that do not have an active deadline. These pods usually include long running pods whose container command is not expected to terminate.

In this exercise we are going to :

  • create a NotTerminating quota of 2 Pods
  • create 2 running Pods
  • attempt to create a third Pod
nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    count/pods: "2"
  scopes:
  - NotTerminating

Create NotTerminating quota and describe it:

kubectl create -f myQuota.yaml
resourcequota/object-counts created


kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Scopes:     NotTerminating
 * Matches all pods that do not have an active deadline. These pods usually include long running pods whose container command is not expected to terminate.
Resource    Used  Hard
--------    ----  ----
count/pods  0     2

Create one Pod and check it gets counted as NotTerminating / running.

kubectl create -f myQuota-Pod-1.yaml
pod/quota-pod-1 created


kubectl describe quota object-counts

Name:       object-counts
Namespace:  default
Scopes:     NotTerminating
 * Matches all pods that do not have an active deadline. These pods usually include long running pods whose container command is not expected to terminate.
Resource    Used  Hard
--------    ----  ----
count/pods  1     2

Is Pod running?

kubectl get po
NAME          READY   STATUS    RESTARTS   AGE
quota-pod-1   1/1     Running   0          10s

NotTerminating correctly counts this running Pod against NotTerminating quota.

Create second running Pod:

kubectl create -f myQuota-Pod-2.yaml
pod/quota-pod-2 created

kubectl get po

NAME          READY   STATUS    RESTARTS   AGE
quota-pod-1   1/1     Running   0          20s
quota-pod-2   1/1     Running   0          4s

Show status of quota counters:

kubectl describe quota object-counts
Name:       object-counts
Namespace:  default
Scopes:     NotTerminating
 * Matches all pods that do not have an active deadline. These pods usually include long running pods whose container command is not expected to terminate.
Resource    Used  Hard
--------    ----  ----
count/pods  2     2

Both running Pods count against NotTerminating quota.

Attempt to run a third Pod should result in error:

kubectl create -f myQuota-Pod-3.yaml
Error from server (Forbidden): error when creating "myQuota-Pod-3.yaml": pods "quota-pod-3" is forbidden: exceeded quota: object-counts, requested: count/pods=1, used: count/pods=2, limited: count/pods=2

Delete objects:

kubectl delete quota/object-counts
resourcequota "object-counts" deleted

kubectl delete -f myQuota-Pod-1.yaml
pod "quota-pod-1" deleted

kubectl delete -f myQuota-Pod-2.yaml
pod "quota-pod-2" deleted

6) Quotas Need Value in YAML Spec

https://kubernetes.io/docs/concepts/policy/resource-quotas/#requests-vs-limits

If the quota has a value specified for requests.cpu or requests.memory, then it requires that every incoming container makes an explicit request for those resources.

If the quota has a value specified for limits.cpu or limits.memory, then it requires that every incoming container specifies an explicit limit for those resources

To see this in action we define a ResourceQuota with requests.cpu and limits.cpu.

nano myQuota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-counts
spec:
  hard:
    requests.cpu: "1000m"
    limits.cpu: "2000m"
    count/pods: "2"

Create the Quota

kubectl create -f myQuota.yaml

resourcequota/count-quotas created

Describe quota:

kubectl describe quota object-counts

Name:         object-counts
Namespace:    default
Resource      Used  Hard
--------      ----  ----
count/pods    0     2
limits.cpu    0     2
requests.cpu  0     1

Now we attempt to create a Pod without specifying CPU limits and resources:

nano myQuota-Pod-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: quota-pod-1
spec:
  containers:
  - name: quota-container
    image: busybox
    imagePullPolicy: IfNotPresent
    
    command: ['sh', '-c', 'echo Pod is Running ; sleep 3600']
                 
  restartPolicy: Never
  terminationGracePeriodSeconds: 0

Create the Pod.

kubectl create -f myQuota-Pod-1.yaml

Error from server (Forbidden): error when creating "myQuota-Pod-1.yaml": pods "quota-pod-1" is forbidden: failed quota: object-counts: must specify limits.cpu,requests.cpu

Error message is exactly as expected.

Delete Quota

kubectl delete -f myQuota.yaml

resourcequota "counts-quota" deleted

7) Resource Quota Per PriorityClass

The Kubernetes documentation provides a complete example of this ...

https://kubernetes.io/docs/concepts/policy/resource-quotas/#resource-quota-per-priorityclass

Just follow those instructions.

8) Storage Resource Quota

https://kubernetes.io/docs/concepts/policy/resource-quotas/#storage-resource-quota

You can limit the total sum of storage resources that can be requested in a given namespace.

In addition, you can limit consumption of storage resources based on associated storage-class.

This tutorial gave several repetitively similar exercises for several differences Kubernetes resources.

Once you need quotas on your storage resources you should be able to easily apply your knowledge learnt here.

9) Quota and Cluster Capacity

From https://kubernetes.io/docs/concepts/policy/resource-quotas/#quota-and-cluster-capacity

ResourceQuotas are independent of the cluster capacity. They are expressed in absolute units.

So, if you add nodes to your cluster, this does not automatically give each namespace the ability to consume more resources.

Note that resource quota divides up aggregate cluster resources, but it creates no restrictions around nodes : pods from several namespaces may run on the same node .

10) Start Using Quotas

If you only start using quotas months after starting to use Kubernetes you will have several different Kubernetes objects that already exists on your nodes.

You may then create quotas that are too small for even your current environment.

One way around this is to create temporary quotas that are 100 or 1000 more than your planned values.

You can then immediately run kubectl describe quota to instantly see accurate current counts for all Kubernetes objects you need quotas for. Then just adjust your quota values downwards as needed.

If you create quotas below your current usage that quota will be created without giving an error message.

This is bad, since if for example Pods are under specified, a Pod delete and recreate will fail since you are over quota already.

Therefore every time before you create a new quota run kubectl describe quota to see current resource usage levels.

After you create that quota run kubectl describe quota immediately to check its values are correct.

Container Computing Tech Share Container Service Kubernetes Tutorials Resource Quotas
0 0 0
Share on

Read previous post:

Kubernetes CronJobs - Part 2: Parallelism

Read next post:

How to Manage MySQL Database with Commands

Alibaba Clouder

2,605 posts | 747 followers

You may also like

Comments

Alibaba Clouder

2,605 posts | 747 followers

Related Products

More Posts by Alibaba Clouder

See All