By Alwyn Botha, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.
This tutorial teaches you how to assign memory resources and limits to containers with Kubernetes on Alibaba Cloud. This tutorial will cover two related topics:
We will first define some Pods with our self-imposed limits specified. Then we will test how those limits are enforced.
During the last part of this tutorial we will use LimitRanges. Those limits define memory limits for Pod declaration as well as separate limits for runtime memory use. We will also test how those limits are enforced.
Once you have completed this tutorial, you are invited to create your own LimitRanges and Pods to test your understanding of this topic.
This tutorial will cover the following topics:
Note the syntax below. This is how we define:
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The Bench Pod is Running ; sleep 3600']
resources:
limits:
memory: "100Mi"
requests:
memory: "50Mi"
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Now we investigate how Kubernetes deals with Pods exceeding those limits.
We use an image: mytutorials/centos:bench, that I created and uploaded to the docker hub.
It contains a simple CentOS 7 base operating system. It also includes stress, a benchmark and stress test application.
Syntax:
stress --vm 1 --vm-bytes 50M --vm-hang 10
If you do not use --vm-hang the Pod will continuously use 100% CPU since it will continuously re-allocate around 50 MB of ram with no pause between re-allocations.
We do not want to 100 % stress out the CPU, this hang/wait functionality causes the stress test to use near-zero CPU.
We use the command kubectl exec to ssh into the Pod. Here we can run commands at the Linux shell.
kubectl exec mybench-pod -i -t -- /bin/bash
# stress --vm 1 --vm-bytes 50M --vm-hang 10
Truncated output of top command I ran in another shell.
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
21149 root 20 0 57.2m 50.4m 0.0 2.7 0:00.11 S stress --vm 1 --vm-bytes 50M --vm-hang 10
21148 root 20 0 7.1m 0.9m 0.0 0.0 0:00.00 S stress --vm 1 --vm-bytes 50M --vm-hang 10
Container can easily handle allocation of 50MB RAM.
# stress --vm 1 --vm-bytes 90M --vm-hang 10
Container can easily handle allocation of 90MB RAM.
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
21875 root 20 0 97.2m 90.5m 0.0 4.9 0:00.02 S stress --vm 1 --vm-bytes 90M --vm-hang 10
21874 root 20 0 7.1m 0.9m 0.0 0.0 0:00.00 S stress --vm 1 --vm-bytes 90M --vm-hang 10
stress --vm 1 --vm-bytes 95M --vm-hang 10
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
22143 root 20 0 102.2m 95.3m 0.0 5.1 0:00.02 S stress --vm 1 --vm-bytes 95M --vm-hang 10
22142 root 20 0 7.1m 0.6m 0.0 0.0 0:00.00 S stress --vm 1 --vm-bytes 95M --vm-hang 10
Container can easily handle allocation of 95MB RAM.
[root@mybench-pod /]# stress --vm 1 --vm-bytes 97M --vm-hang 10
stress: info: [78] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [78](415) <-- worker 79 got signal 9
stress: WARN: [78](417) now reaping child worker processes
stress: FAIL: [78](451) failed run completed in 0s
[root@mybench-pod /]# exit
Allocating 97MB RAM fails - our CentOS container needs around 3 MB to run.
We went over our 100Mi RAM limit.
Note that only this stress thread crashed, not the whole container and not the whole Pod.
Output of kubectl describe myapp-pod with ONLY important status fields shown:
Status: Running
State: Running
Started: Thu, 10 Jan 2019 07:47:43 +0200
Ready: True
Restart Count: 0
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Events:
Type Reason Age From Message
Normal Scheduled 12m default-scheduler Successfully assigned default/mybench-pod to minikube
Normal Pulled 12m kubelet, minikube Container image "centos:bench" already present on machine
Normal Created 12m kubelet, minikube Created container
Normal Started 12m kubelet, minikube Started container
Note that the Pod is still running. CentOS 7 is still running and we can still ssh into Pod and still do typical Linux shell work.
First demo complete, delete our Pod:
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "myapp-pod" force deleted
Now we have 2 containers with memory limits. Detail in YAML file below:
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container-1
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo mybench-container-1 is Running ; sleep 3600']
resources:
limits:
memory: "100Mi"
requests:
memory: "50Mi"
- name: mybench-container-2
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo mybench-container-2 is Running ; sleep 3600']
resources:
limits:
memory: "100Mi"
requests:
memory: "50Mi"
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Allocate 88MB in container number 2.
kubectl exec mybench-pod -c mybench-container-2 -i -t -- /bin/bash
[root@mybench-pod /]# stress --vm 1 --vm-bytes 88M --vm-hang 10
stress: info: [22] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
Open another shell:
Allocate 95MB in container number 1.
kubectl exec mybench-pod -c mybench-container-1 -i -t -- /bin/bash
[root@mybench-pod /]# stress --vm 1 --vm-bytes 95M --vm-hang 10
stress: info: [23] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
Each container has its own max RAM limit of 100Mi.
top output below shows both containers running.
PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND
26346 root 20 0 102.2m 95.4m 0.8 5.1 0:00.04 S stress --vm 1 --vm-bytes 95M --vm-hang 10
26244 root 20 0 95.2m 88.4m 0.0 4.7 0:00.01 S stress --vm 1 --vm-bytes 88M --vm-hang 10
26345 root 20 0 7.1m 0.8m 0.0 0.0 0:00.00 S stress --vm 1 --vm-bytes 95M --vm-hang 10
26243 root 20 0 7.1m 0.8m 0.0 0.0 0:00.00 S stress --vm 1 --vm-bytes 88M --vm-hang 10
Press control c in any of your shells, and allocate 120MB RAM. It crashes since it exceeds 100Mi limit.
[root@mybench-pod /]# stress --vm 1 --vm-bytes 120M --vm-hang 10
stress: info: [24] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [24](415) <-- worker 25 got signal 9
stress: WARN: [24](417) now reaping child worker processes
stress: FAIL: [24](451) failed run completed in 0s
Type exit to exit this shell.
Press control c in the other shell, type exit to exit this shell.
Summary: all containers in a Pod have independent RAM limits. You have fine-grained control of each container independently.
( Not part of this tutorial, but each container can have independent CPU constraints as well. )
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "myapp-pod" force deleted
We have seen that the overall status of Pod is unaffected by failing threads inside it.
Now we will see what happens if a Pod contains a container that upon startup exceeds the RAM limit.
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "10"]
The YAML spec above specifies a command to run upon startup, exceeding its self-imposed / self-declared limit of 100Mi of RAM.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container-1
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "10"]
resources:
limits:
memory: "100Mi"
requests:
memory: "50Mi"
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Immediately afterwards:
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 OOMKilled 0 9s
OOM means out of memory.
Output of kubectl describe mybench-pod with ONLY important status fields shown:
Status: Failed
State: Terminated
Reason: OOMKilled
Exit Code: 1
Started: Thu, 10 Jan 2019 08:30:13 +0200
Finished: Thu, 10 Jan 2019 08:30:14 +0200
Ready: False
Restart Count: 0
We had restartPolicy: Never in the Pod spec.
Once it got killed with OOM Killed, it stayed killed: never restarting.
Investigate what happens with restartPolicy: OnFailure
( Note last line in spec )
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container-1
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "10"]
resources:
limits:
memory: "100Mi"
requests:
memory: "50Mi"
restartPolicy: OnFailure
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Check on Pod status :
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 OOMKilled 1 2s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 CrashLoopBackOff 1 6s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 CrashLoopBackOff 1 12s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 CrashLoopBackOff 1 15s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 OOMKilled 2 24s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 CrashLoopBackOff 2 35s
kubectl get po
NAME READY STATUS RESTARTS AGE
mybench-pod 0/1 OOMKilled 3 45s
restartPolicy: OnFailure faithfully but without success restarts Pod repeatedly.
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "myapp-pod" force deleted
Kubernetes administrators can define RAM limits for their nodes.
These limits are enforced at higher priority over how much RAM your Pod declares and wants to use.
Let's define our first LimitRange : 25Mi RAM as min, 200Mi as max.
nano myRAM-LimitRange.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: my-ram-limit
spec:
limits:
- max:
memory: 200Mi
min:
memory: 25Mi
kubectl create -f myRAM-LimitRange.yaml
MB not supported: You cannot specify RAM in MB, must be in Mi. This is error you will get:
kubectl create -f myRAM-LimitRange.yaml
Error from server (BadRequest): error when creating "myRAM-LimitRange.yaml": LimitRange in version "v1" cannot be handled as a LimitRange: v1.LimitRange.Spec: v1.LimitRangeSpec.Limits: []v1.LimitRangeItem: v1.LimitRangeItem.Min: Max: unmarshalerDecoder: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$', error found in #10 byte of ...|y":"250MB"},"min":{"|..., bigger context ...|fault"},"spec":{"limits":[{"max":{"memory":"250MB"},"min":{"memory":"25MB"},"type":"Container"}]}}
Some namespaces or nodes may be dedicated to vast Pods. No tiny Pods allowed.
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The Bench Pod is Running ; sleep 3600']
resources:
limits:
memory: "100Mi"
requests:
memory: "10Mi"
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
Error from server (Forbidden): error when creating "myBench-Pod.yaml": pods "mybench-pod" is forbidden: minimum memory usage per Container is 25Mi, but request is 10Mi.
Error easy to understand.
Let's request too much RAM:
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The Bench Pod is Running ; sleep 3600']
resources:
limits:
memory: "300Mi"
requests:
memory: "30Mi"
restartPolicy: Never
kubectl create -f myBench-Pod.yaml
Error from server (Forbidden): error when creating "myBench-Pod.yaml": pods "mybench-pod" is forbidden: maximum memory usage per Container is 250Mi, but limit is 300Mi.
Easy to understand.
Let's define a Pod within the allowed RAM ranges:
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The Bench Pod is Running ; sleep 3600']
resources:
limits:
memory: "100Mi"
requests:
memory: "30Mi"
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Success.
If we now use kubectl exec and allocate too much RAM that thread gets killed ( as expected ).
kubectl exec mybench-pod -i -t -- /bin/bash
[root@mybench-pod /]# stress --vm 1 --vm-bytes 50M --vm-hang 10
stress: info: [22] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
^C
[root@mybench-pod /]# stress --vm 1 --vm-bytes 90M --vm-hang 10
stress: info: [24] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
^C
[root@mybench-pod /]# stress --vm 1 --vm-bytes 120M --vm-hang 10
stress: info: [26] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [26](415) <-- worker 27 got signal 9
stress: WARN: [26](417) now reaping child worker processes
stress: FAIL: [26](451) failed run completed in 1s
[root@mybench-pod /]# exit
kubectl delete -f myBench-Pod.yaml --force --grace-period=0
pod "myapp-pod" force deleted
Finished with this limit definition, delete it.
kubectl delete limits/my-ram-limit
limitrange "my-ram-limit" deleted
Previously our LimitRange only defined limits.
We can also use LimitRange to define default limits. These defaults are used when a Pod spec does not specify its own limits.
apiVersion: v1
kind: LimitRange
metadata:
name: my-ram-limit
spec:
limits:
- default:
memory: 150Mi
defaultRequest:
memory: 30Mi
max:
memory: 250Mi
min:
memory: 25Mi
type: Container
The first 5 lines would be more clear if the syntax was:
limits:
- defaultLimit:
memory: 150Mi
defaultRequest:
memory: 30Mi
Unfortunately the syntax is:
limits:
- default:
memory: 150Mi
defaultRequest:
memory: 30Mi
Let's create our new LimitRange:
kubectl create -f myRAM-LimitRange.yaml
limitrange/my-ram-limit created
Let's let Kubernetes describe what it did:
kubectl describe limits
Name: my-ram-limit
Namespace: default
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory 25Mi 250Mi 30Mi 150Mi -
Perfectly understandable output.
Now we need a Pod that does not specify limits in its spec.
See no limits in spec below:
nano myBench-Pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mybench-pod
spec:
containers:
- name: mybench-container
image: mytutorials/centos:bench
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo The Bench Pod is Running ; sleep 3600']
restartPolicy: Never
Create the Pod.
kubectl create -f myBench-Pod.yaml
pod/mybench-pod created
Output of kubectl describe mybench-pod with ONLY important status fields shown:
Name: mybench-pod
Annotations: kubernetes.io/limit-ranger:
LimitRanger plugin set: memory request for container mybench-container; memory limit for container mybench-container
Status: Running
IP: 172.17.0.6
Containers:
mybench-container:
...
Limits:
memory: 150Mi
Requests:
memory: 30Mi
...
Note the annotation : this Pod spec memory got auto adjusted using a limit-ranger.
At the bottom we can see this Pod has RAM limits based on LimitRange defaults.
We expect that if we use more than 150Mi of RAM that such an allocation will crash.
Let's test it:
kubectl exec mybench-pod -i -t -- /bin/bash
[root@mybench-pod /]# stress --vm 1 --vm-bytes 140M --vm-hang 10
stress: info: [27] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
^C
[root@mybench-pod /]# stress --vm 1 --vm-bytes 160M --vm-hang 10
stress: info: [29] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [29](415) <-- worker 30 got signal 9
stress: WARN: [29](417) now reaping child worker processes
stress: FAIL: [29](451) failed run completed in 0s
[root@mybench-pod /]# exit
As expected: 140MB within limit, no problem, 160MB got killed ( as expected ).
Throughout this tutorial we used these limits in the default namespace.
In a complex development and production environment Pods run in different namespaces.
Namespaces separate Kubernetes resources into logical groupings.
Similarly LimitRanges can be defined to be enforced per namespace.
If you work at a small company with just one namespace the LimitRanges you define will be in the default namespace - applicable to all Pods .
For more information: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
and
https://kubernetes.io/docs/tasks/administer-cluster/namespaces-walkthrough/
Some online Kubernetes documentation stress the underlying architecture when describing any Kubernetes topic.
Nearly everything in Kubernetes is an API object: namespaces, Pods, LimitRanges.
Real people say: I used a LimitRange to define RAM limits for Pods. I used kubectl to create it.
Real people do not say: I used a Kubernetes API LimitRange object to define RAM limits for my Kubernetes API Pod object. I used the RESTful interface of kubectl to call the apiserver to create those API objects in etcd.
Unless your work focus is the Kubernetes underlying architecture ( writing API calls ) you may safely skip references to API.
Knowing that etcd is the 'highly-available key value store used as Kubernetes' backing store for all cluster API resources data' adds nothing to your understanding of RAM limits.
This tutorial does not provide an exhaustive list of all the possibilities of using limits. Instead, it provides you with sufficient information for you to understand the its underlying logic. In summary, we learned that:
You can now define Pods and LimitRanges and accurately predict what will happen in each Pod with any size memory allocation.
Note: If your Pods do not define limits and there are no LimitRanges then one container can use all the RAM on your Kubernetes node.
Exercise: define your own LimitRange and one Pod. Repeatedly adjust your Pods' limits around, inside and outside those ranges. Create the Pods and do similar stress tests to test you understanding.
Each such test cycle could be done in 2 minutes. If, after 15 minutes, all your predicted expectations are correct, you may stop and congratulate yourself.
2,599 posts | 763 followers
FollowAlibaba Clouder - May 21, 2019
Alibaba Cloud Blockchain Service Team - December 26, 2018
Alibaba Cloud Native Community - December 29, 2023
Alibaba Developer - February 26, 2020
Alibaba Container Service - March 12, 2021
Alibaba Container Service - September 14, 2024
2,599 posts | 763 followers
FollowAlibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn MoreA secure image hosting platform providing containerized image lifecycle management
Learn MoreElastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreMore Posts by Alibaba Clouder