Kubernetes allows you to deploy multiple types of containerized applications on the same host. Applications with different priorities share the L3 cache (last level cache) and memory bandwidth that are provided by the host. ack-koordinator enables the isolation of L3 cache and memory bandwidth for different applications. This ensures the quality of service (QoS) of high-priority applications when applications compete for resources. This topic describes how to enable resource isolation for applications with different priorities by controlling the L3 cache and using the Memory Bandwidth Allocation (MBA) feature.
Prerequisites
- The Helm component is installed and the component version is 3.0 or later. For more information about how to update the Helm component, see [Component Updates] Update Helm V2 to V3 or How do I manually update Helm?.
- The host that you use is an Elastic Compute Service (ECS) bare metal instance and the CPU model of the host supports the Resource Director Technology (RDT). For more information, see Overview and intel-cmt-cat.
- The ack-koordinator(ack-slo-manager) component is installed and the component version is 0.8.0 or later. For more information about how to install ack-koordinator, see ack-koordinator (FKA ack-slo-manager).
Background information
To make full use of computing resources, workloads of different priorities are usually deployed on the same node. For example, a latency-sensitive (LS) workload (with a high priority) and a best-effort (BE) workload (with a low priority) can be deployed on the same node. However, this may cause these workloads to compete for computing resources. If you do not enable resource isolation, workloads of different priorities may compete for computing resources such as the L3 cache and memory bandwidth. As a result, the computing resources that are allocated to LS workloads may be insufficient and the QoS of LS workloads is degraded.
The RDT enables resource isolation for workloads of different priorities by limiting the L3 cache and memory bandwidth that BE workloads can use. This ensures the QoS of LS workloads in scenarios where workloads of different priorities are deployed. For more information about the RDT, see resource-director-technology.
Before you begin
Before you use the L3 cache and MBA to enable resource isolation, you must enable the RDT feature of the kernel. To enable the RDT feature of the kernel, perform the following steps:
- Run the following command to check whether the RDT feature of the kernel is enabled:
cat /proc/cmdline
Expected output:
# Only the settings that are related to the RDT feature are displayed in the BOOT_IMAGE field. Irrelevant settings are omitted. BOOT_IMAGE=... rdt=cmt,l3cat,l3cdp,mba
If the output contains the
l3cat
andmba
keywords, the RDT feature is enabled. If the output does not contain the keywords, you can perform the following steps to enable the RDT feature. - Enable the RDT feature of the kernel.
Procedure
After you enable the RDT feature of the kernel, perform the following steps to isolate the L3 cache and memory bandwidth that are used by different workloads:
- Create a configmap.yaml file with the following YAML template. Set
enable
totrue
to isolate the L3 cache and memory bandwidth that are used by BE workloads.apiVersion: v1 kind: ConfigMap metadata: name: ack-slo-config namespace: kube-system data: resource-qos-config: | { "clusterStrategy": { "beClass": { "resctrlQOS": { "enable": true } } } }
- Check whether the
ack-slo-config
ConfigMap exists in thekube-system
namespace.- If the
ack-slo-config
ConfigMap exists, we recommend that you run the kubectl patch command to update the ConfigMap. This avoids changing other settings in the ConfigMap.kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"
- If the
ack-slo-config
ConfigMap does not exist, run the kubectl patch command to create a ConfigMap named ack-slo-config:kubectl apply -f configmap.yaml
- If the
- Create a pod-demo.yaml file with the following YAML template. Set the QoS class of the pod to
BE
to limit the L3 cache and memory bandwidth that the pod can use.apiVersion: v1 kind: Pod metadata: name: pod-demo labels: koordinator.sh/qosClass: 'BE' spec: containers: - name: pod-demo image: polinux/stress resources: requests: cpu: 1 memory: "50Mi" limits: cpu: 1 memory: "1Gi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
- Run the following command to deploy the pod-demo application in the cluster:
kubectl apply -f pod-demo.yaml
- Configure advanced parameters based on the following YAML template.
You can enable fine-grained isolation of the L3 cache and memory bandwidth based on the QoS classes of workloads.
# Example of the ack-slo-config ConfigMap. apiVersion: v1 kind: ConfigMap metadata: name: ack-slo-config namespace: kube-system data: resource-qos-config: | { "clusterStrategy": { "lsClass": { "resctrlQOS": { "enable": true, "catRangeEndPercent": 100, "mbaPercent": 100 } }, "beClass": { "resctrlQOS": { "enable": true, "catRangeEndPercent": 30, "mbaPercent": 100 } } } }
The following table describes some parameters that are specified in the preceding code block.
Parameter Type Valid value Description enable
Boolean true
false
true
: enables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.false
: disables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
catRangeEndPercent
Int 0~100 The percentage of the L3 cache that can be used by the workloads of the corresponding QoS class. Unit: %. The default value for workloads of the LS class is 100
. The default value for workloads of the BE class is30
.mbaPercent
Int 0~100 The percentage of the memory bandwidth that can be used by the workloads of the corresponding QoS class. Unit: %. You must set the value to a multiple of 10. The default values for the workloads of the LS class and BE class are both 100
.