Kubernetes allows you to deploy multiple types of containerized applications on the same host. Applications with different priorities share the L3 cache (last level cache) and memory bandwidth that are provided by the host. ack-koordinator enables the isolation of L3 cache and memory bandwidth for different applications. This ensures the quality of service (QoS) of high-priority applications when applications compete for resources. This topic describes how to enable resource isolation for applications with different priorities by controlling the L3 cache and using the Memory Bandwidth Allocation (MBA) feature.
Background information
To make full use of computing resources, workloads of different priorities are usually deployed on the same node. For example, a latency-sensitive (LS) workload (with a high priority) and a best-effort (BE) workload (with a low priority) can be deployed on the same node. However, this may cause these workloads to compete for computing resources. If you do not enable resource isolation, workloads of different priorities may compete for computing resources such as the L3 cache and memory bandwidth. As a result, the computing resources that are allocated to LS workloads may be insufficient and the QoS of LS workloads is degraded.
The RDT enables resource isolation for workloads of different priorities by limiting the L3 cache and memory bandwidth that BE workloads can use. This ensures the QoS of LS workloads in scenarios where workloads of different priorities are deployed. For more information about the RDT, see resource-director-technology.
Prerequisites
The Helm component is installed and the component version is 3.0 or later. For more information about how to update the Helm component, see [Component Updates] Update Helm V2 to V3 or How do I manually update Helm?
The host that you use is an Elastic Compute Service (ECS) bare metal instance and the CPU model of the host supports the Resource Director Technology (RDT). For more information, see Overview and intel-cmt-cat.
The ack-koordinator(ack-slo-manager) component is installed and the component version is 0.8.0 or later. For more information about how to install ack-koordinator, see ack-koordinator (FKA ack-slo-manager).
Before you begin
Before you use the L3 cache and MBA to enable resource isolation, you must enable the RDT feature of the kernel. To enable the RDT feature of the kernel, perform the following steps:
Run the following command to check whether the RDT feature of the kernel is enabled:
cat /proc/cmdline
Expected output:
# Only the settings that are related to the RDT feature are displayed in the BOOT_IMAGE field. Irrelevant settings are omitted. BOOT_IMAGE=... rdt=cmt,l3cat,l3cdp,mba
If the output contains the
l3cat
andmba
keywords, the RDT feature is enabled. If the output does not contain the preceding keywords, you can perform the following steps to enable the RDT feature.Enable the RDT feature of the kernel.
Modify the /etc/default/grub file.
Configure the
GRUB_CMDLINE_LINUX
field to enable the RDT feature.# Only the settings that are related to the RDT feature are displayed in the GRUB_CMDLINE_LINUX field. Irrelevant settings are omitted. GRUB_CMDLINE_LINUX="... rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp,mba"
ImportantSeparate the settings that are related to the RDT feature and other settings with a space character.
Run the following command to update the grub.cfg file:
# Set the path to the actual path of the file. sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Run the following command to restart the node:
sudo systemctl reboot
Procedure
After you enable the RDT feature of the kernel, perform the following steps to isolate the L3 cache and memory bandwidth that are used by different workloads:
Create a configmap.yaml file with the following YAML template.
Set enable to
true
to isolate the L3 cache and memory bandwidth that are used by BE workloads.apiVersion: v1 kind: ConfigMap metadata: name: ack-slo-config namespace: kube-system data: resource-qos-config: | { "clusterStrategy": { "beClass": { "resctrlQOS": { "enable": true } } } }
Check whether the
ack-slo-config
ConfigMap exists in thekube-system
namespace.If the
ack-slo-config
ConfigMap exists, we recommend that you run the kubectl patch command to update the ConfigMap. This avoids changing other settings in the ConfigMap.kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"
If the
ack-slo-config
ConfigMap does not exist, run the following command to create a ConfigMap named ack-slo-config:kubectl apply -f configmap.yaml
Create a file named pod-demo.yaml and copy the following content to the file:
Set the QoS class of the pod to
BE
to limit the L3 cache and memory bandwidth that the pod can use.apiVersion: v1 kind: Pod metadata: name: pod-demo labels: koordinator.sh/qosClass: 'BE' spec: containers: - name: pod-demo image: polinux/stress resources: requests: cpu: 1 memory: "50Mi" limits: cpu: 1 memory: "1Gi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
Run the following command to deploy the pod-demo pod in the cluster:
kubectl apply -f pod-demo.yaml
Configure advanced parameters based on the following YAML template.
You can enable fine-grained isolation of the L3 cache and memory bandwidth based on the QoS classes of workloads.
# Example of the ack-slo-config ConfigMap. apiVersion: v1 kind: ConfigMap metadata: name: ack-slo-config namespace: kube-system data: resource-qos-config: | { "clusterStrategy": { "lsClass": { "resctrlQOS": { "enable": true, "catRangeEndPercent": 100, "mbaPercent": 100 } }, "beClass": { "resctrlQOS": { "enable": true, "catRangeEndPercent": 30, "mbaPercent": 100 } } } }
The following table describes some of the parameters.
Parameter
Type
Valid value
Description
enable
Boolean
true
false
true
: enables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.false
: disables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
catRangeEndPercent
Int
0~100
The percentage of the L3 cache that can be used by the workloads of the corresponding QoS class. Unit: %. The default value for workloads of the LS class is
100
. The default value for workloads of the BE class is30
.mbaPercent
Int
0~100
The percentage of the memory bandwidth that can be used by the workloads of the corresponding QoS class. Unit: %. You must set the value to a multiple of 10. The default values for the workloads of the LS class and BE class are both
100
.