Kubernetes allows you to deploy multiple types of containerized applications on the same host. Applications with different priorities share the L3 cache (last level cache) and memory bandwidth that are provided by the host. ack-koordinator enables the isolation of L3 cache and memory bandwidth for different applications. This ensures the quality of service (QoS) of high-priority applications when applications compete for resources. This topic describes how to enable resource isolation for applications with different priorities by controlling the L3 cache and using the Memory Bandwidth Allocation (MBA) feature.

Prerequisites

Background information

To make full use of computing resources, workloads of different priorities are usually deployed on the same node. For example, a latency-sensitive (LS) workload (with a high priority) and a best-effort (BE) workload (with a low priority) can be deployed on the same node. However, this may cause these workloads to compete for computing resources. If you do not enable resource isolation, workloads of different priorities may compete for computing resources such as the L3 cache and memory bandwidth. As a result, the computing resources that are allocated to LS workloads may be insufficient and the QoS of LS workloads is degraded.

The RDT enables resource isolation for workloads of different priorities by limiting the L3 cache and memory bandwidth that BE workloads can use. This ensures the QoS of LS workloads in scenarios where workloads of different priorities are deployed. For more information about the RDT, see resource-director-technology.

Before you begin

Before you use the L3 cache and MBA to enable resource isolation, you must enable the RDT feature of the kernel. To enable the RDT feature of the kernel, perform the following steps:

  1. Run the following command to check whether the RDT feature of the kernel is enabled:
    cat /proc/cmdline

    Expected output:

    # Only the settings that are related to the RDT feature are displayed in the BOOT_IMAGE field. Irrelevant settings are omitted. 
    BOOT_IMAGE=... rdt=cmt,l3cat,l3cdp,mba

    If the output contains the l3cat and mba keywords, the RDT feature is enabled. If the output does not contain the keywords, you can perform the following steps to enable the RDT feature.

  2. Enable the RDT feature of the kernel.
    1. Modify the /etc/default/grub file.
      Configure the GRUB_CMDLINE_LINUX field to enable the RDT feature.
      # Only the settings that are related to the RDT feature are displayed in the GRUB_CMDLINE_LINUX field. Irrelevant settings are omitted. 
      GRUB_CMDLINE_LINUX="... rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp,mba" 
      Important Separate the settings that are related to the RDT feature and other settings with a space character.
    2. Run the following command to update the grub.cfg file:
      # Set the path to the actual path of the file. 
      sudo grub2-mkconfig -o /boot/grub2/grub.cfg
    3. Run the following command to restart the node:
      sudo systemctl reboot

Procedure

After you enable the RDT feature of the kernel, perform the following steps to isolate the L3 cache and memory bandwidth that are used by different workloads:

  1. Create a configmap.yaml file with the following YAML template.
    Set enable to true to isolate the L3 cache and memory bandwidth that are used by BE workloads.
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ack-slo-config
      namespace: kube-system
    data:
      resource-qos-config: |
        {
          "clusterStrategy": {
            "beClass": {
              "resctrlQOS": {
                "enable": true
              }
            }
          }
        }
  2. Check whether the ack-slo-config ConfigMap exists in the kube-system namespace.
    • If the ack-slo-config ConfigMap exists, we recommend that you run the kubectl patch command to update the ConfigMap. This avoids changing other settings in the ConfigMap.
      kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"
    • If the ack-slo-config ConfigMap does not exist, run the kubectl patch command to create a ConfigMap named ack-slo-config:
      kubectl apply -f configmap.yaml
  3. Create a pod-demo.yaml file with the following YAML template.
    Set the QoS class of the pod to BE to limit the L3 cache and memory bandwidth that the pod can use.
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-demo
      labels:
        koordinator.sh/qosClass: 'BE'
    spec:
      containers:
      - name: pod-demo
        image: polinux/stress
        resources:
          requests:
            cpu: 1
            memory: "50Mi"
          limits:
            cpu: 1
            memory: "1Gi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
  4. Run the following command to deploy the pod-demo application in the cluster:
    kubectl apply -f pod-demo.yaml
  5. Configure advanced parameters based on the following YAML template.

    You can enable fine-grained isolation of the L3 cache and memory bandwidth based on the QoS classes of workloads.

    # Example of the ack-slo-config ConfigMap. 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ack-slo-config
      namespace: kube-system
    data:
      resource-qos-config: |
        {
          "clusterStrategy": {
            "lsClass": {
              "resctrlQOS": {
                "enable": true,
                "catRangeEndPercent": 100,
                "mbaPercent": 100
              }
            },
            "beClass": {
              "resctrlQOS": {
                "enable": true,
                "catRangeEndPercent": 30,
                "mbaPercent": 100
              }
            }
          }
        }

    The following table describes some parameters that are specified in the preceding code block.

    ParameterTypeValid valueDescription
    enableBoolean
    • true
    • false
    • true: enables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
    • false: disables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
    catRangeEndPercentInt0~100The percentage of the L3 cache that can be used by the workloads of the corresponding QoS class. Unit: %. The default value for workloads of the LS class is 100. The default value for workloads of the BE class is 30.
    mbaPercentInt0~100The percentage of the memory bandwidth that can be used by the workloads of the corresponding QoS class. Unit: %. You must set the value to a multiple of 10. The default values for the workloads of the LS class and BE class are both 100.