Kubernetes allows you to deploy multiple types of containerized applications on the same host. Applications with different priorities share the L3 cache (last level cache) and memory bandwidth that are provided by the host. ack-slo-manager enables the isolation of L3 cache and memory bandwidth for different applications. This ensures the quality of service (QoS) of high-priority applications when applications compete for resources. This topic describes how to enable resource isolation for applications with different priorities by controlling the L3 cache and using the Memory Bandwidth Allocation (MBA) feature.

Prerequisites

  • The Helm component is installed and the component version is 3.0 or later. For more information about how to update the Helm component, see [Component Upgrades] Upgrade Helm V2 to V3 or How do I manually update Helm?.
  • The host that you use is an Elastic Compute Service (ECS) bare metal instance and the CPU model of the host supports the Resource Director Technology (RDT). For more information, see Overview and intel-cmt-cat.
  • The ack-slo-manager component is installed and the component version is 0.4.0 or later. For more information about how to install ack-slo-manager, see Usage notes.
    Note ack-slo-manager is upgraded and optimized based on resource-controller. You must uninstall resource-controller after you install ack-slo-manager. For more information about how to uninstall a component, see Manage system components.

Background information

To make full use of computing resources, workloads of different priorities are usually deployed on the same node. For example, a latency-sensitive (LS) workload (with a high priority) and a best-effort (BE) workload (with a low priority) can be deployed on the same node. However, this may cause these workloads to compete for computing resources. If you do not enable resource isolation, workloads of different priorities may compete for computing resources such as the L3 cache and memory bandwidth. As a result, the computing resources that are allocated to LS workloads may be insufficient and the QoS of LS workloads is degraded.

The RDT enables resource isolation for workloads of different priorities by limiting the L3 cache and memory bandwidth that BE workloads can use. This ensures the QoS of LS workloads in scenarios where workloads of different priorities are deployed. For more information about the RDT, see resource-director-technology.

Preparations

Before you use the L3 cache and MBA to enable resource isolation, you must enable the RDT feature of the kernel. To enable the RDT feature of the kernel, perform the following steps:

  1. Run the following command to check whether the RDT feature of the kernel is enabled:
    cat /proc/cmdline

    Expected output:

    # Only the settings that are related to the RDT feature are displayed in the BOOT_IMAGE field. Irrelevant settings are omitted. 
    BOOT_IMAGE=... rdt=cmt,l3cat,l3cdp,mba

    If the output contains the l3cat and mba keywords, the RDT feature is enabled. If the output does not contain the keywords, you can perform the following steps to enable the RDT feature.

  2. Enable the RDT feature of the kernel.
    1. Modify the /etc/default/grub file.
      Configure the GRUB_CMDLINE_LINUX field to enable the RDT feature.
      # Only the settings that are related to the RDT feature are displayed in the GRUB_CMDLINE_LINUX field. Irrelevant settings are omitted. 
      GRUB_CMDLINE_LINUX="... rdt=cmt,mbmtotal,mbmlocal,l3cat,l3cdp,mba" 
      Notice Separate the settings that are related to the RDT feature and other settings with a space character.
    2. Run the following command to update the grub.cfg file:
      # Set the path to the actual path of the file. 
      grub2-mkconfig -o /boot/grub2/grub.cfg
    3. Run the following command to restart the node:
      systemctl reboot

Procedure

After you enable the RDT feature of the kernel, perform the following steps to isolate the L3 cache and memory bandwidth that are used by different workloads:

  1. Create a configmap.yaml file with the following YAML template.
    Set enable to true to isolate the L3 cache and memory bandwidth that are used by BE workloads.
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ack-slo-manager-config
      namespace: kube-system
    data:
      resource-qos-config: |
        {
          "clusterStrategy": {
            "beClass": {
              "resctrlQOS": {
                "enable": true
              }
            }
          }
        }
  2. Run the following command to update the ConfigMap.
    To avoid changing other settings in the ConfigMap, we recommend that you run the kubectl patch command to update the ConfigMap.
    kubectl patch cm -n kube-system ack-slo-manager-config --patch "$(cat configmap.yaml)"
  3. Create a pod-demo.yaml file with the following YAML template.
    Set the QoS class of the pod to BE to limit the L3 cache and memory bandwidth that the pod can use.
    apiVersion: v1
    kind: Pod
    metadata:
      name: pod-demo
      labels:
        name: pod-demo
      annotations:
        alibabacloud.com/qosClass: 'BE' 
    spec:
      containers:
      - name: pod-demo
        image: polinux/stress
        resources:
          requests:
            cpu: 1
            memory: "50Mi"
          limits:
            cpu: 1
            memory: "1Gi"
        command: ["stress"]
        args: ["--vm", "1", "--vm-bytes", "256M", "-c", "2", "--vm-hang", "1"]
  4. Run the following command to deploy the pod-demo application in the cluster:
    kubectl apply -f pod-demo.yaml
  5. Configure advanced parameters based on the following YAML template.

    You can enable fine-grained isolation of the L3 cache and memory bandwidth based on the QoS classes of workloads.

    # Example of the ack-slo-manager-config ConfigMap. 
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: ack-slo-manager-config
      namespace: kube-system
    data:
      resource-qos-config: |
        {
          "clusterStrategy": {
            "lsClass": {
              "resctrlQOS": {
                "enable": true,
                "catRangeEndPercent": 100,
                "mbaPercent": 100
              }
            },
            "beClass": {
              "resctrlQOS": {
                "enable": true,
                "catRangeEndPercent": 30,
                "mbaPercent": 100
              }
            }
          }
        }

    The following table describes some parameters that are specified in the preceding code block.

    Parameter Data type Valid value Description
    enable Boolean
    • true
    • false
    • true: enables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
    • false: disables the isolation of the L3 cache and memory bandwidth for workloads in the cluster.
    catRangeEndPercent Int 0~100 The percentage of the L3 cache that can be used by the workloads of the corresponding QoS class. Unit: %. The default value for workloads of the LS class is 100. The default value for workloads of the BE class is 30.
    mbaPercent Int 0~100 The percentage of the memory bandwidth that can be used by the workloads of the corresponding QoS class. Unit: %. You must set the value to a multiple of 10. The default values for the workloads of the LS class and BE class are both 100.
    Note ack-slo-manager fully supports the configuration mode of the resource-controller component. We recommend that you use the new configuration mode. To use the original configuration mode, Submit a ticket.