Offload Encryption & Compression to Intel QAT on ACK to Reduce CPU Load - Container Service for Kubernetes

Web servers, API gateways, and databases that handle high volumes of Transport Layer Security (TLS) traffic consume significant CPU capacity on cryptographic operations. Intel® QuickAssist Technology (QAT) is a hardware accelerator built into Intel® Xeon® Scalable processors that offloads symmetric and asymmetric encryption and decryption, compression, and decompression from CPU cores to dedicated hardware—reducing CPU utilization and improving throughput.

ack-qat-deviceplugin, developed from the OpenAnolis intel-accel-plugin-qat open source project, exposes QAT devices on eighth-generation Elastic Compute Service (ECS) Bare Metal Instances (Intel Sapphire Rapids) as on-demand Kubernetes resources. Applications such as NGINX and Envoy request QAT devices via standard Kubernetes resource limits, offloading TLS and GNU zip (Gzip) operations from the CPU.

QAT resource naming

After deploying ack-qat-deviceplugin, each node exposes QAT capacity using the resource name qat.intel.com/cy{N}_dc{N}, where:

cy{N} is the number of work queues for encryption and decryption
dc{N} is the number of work queues for compression and decompression

For example, qat.intel.com/cy2_dc2 means each QAT virtual function (VF) has two work queues for cryptographic operations and two for compression. For an ecs.ebmg8i.48xlarge instance, each QAT device has two work queues for encryption and decryption and two work queues for compression and decompression. Use this resource name in pod limits to allocate QAT devices to your workloads. The exact name depends on your hardware configuration and the setup.vf_per_pf setting.

Limitations

Instance type: Use ECS Bare Metal Instances in the ecs.ebmg8i or ecs.ebmc8i instance families. The examples in this topic use ecs.ebmg8i.48xlarge. These instance families are only available in select regions. To check availability, see Instance types available for each region.
Operating system: Alibaba Cloud Linux UEFI 3.2104 Security Enhanced.
Conflicting device plugins: If a device plugin from another provider is already installed to expose QAT devices, uninstall it before proceeding.

Prerequisites

Before you begin, ensure that you have:

Step 1: Configure nodes

Before using QAT devices for the first time, update the kernel parameters on your nodes and reboot.

Important

To avoid re-running this script each time a node is added, create a dedicated node pool for ecs.ebmg8i or ecs.ebmc8i instances and add this script to the user data field of the node pool—not to the pre-defined custom data field. Adding it to pre-defined custom data prevents nodes from joining the cluster. For details, see Create a node pool.

yum install kernel-0:5.10.134-16.1.al8.x86_64 -y
if [ $? -ne 0 ];then
    echo "Error: yum update failed"
fi

yum install kernel-modules -y
if [ $? -ne 0 ];then
    echo "Error: yum install kernel-modules failed"
fi

kernel_path=$(grubby --default-kernel)
kernel_args="intel_iommu=on iommu=pt"
grubby --update-kernel=$kernel_path --args=$kernel_args

reboot -f

Step 2: Deploy ack-qat-deviceplugin

Install ack-qat-deviceplugin using Helm:

Parameter	Required	Description	Default
`regionId`	No	Region where the cluster resides. When set, the chart is pulled over the internal network.	None
`setup.vf_per_pf`	No	Number of virtual functions (VFs) to create per physical function (PF). Valid values: 0–16.	`16`
`setup.enabled_mode`	No	Driver mode for QAT devices. Valid values: `sym` (symmetric encryption and decryption), `asym` (asymmetric encryption and decryption), `dc` (compression and decompression). Specify up to two modes, separated by semicolons. For example, `asym;dc` enables both asymmetric crypto and compression. For the full parameter reference, see sysfs-driver-qat.	`asym;dc`

helm install ack-qat-deviceplugin https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/pre/charts-incubator/ack-qat-deviceplugin-0.1.2.tgz \
  --set regionId="cn-beijing" \
  --set setup.vf_per_pf="16" \
  --set setup.enabled_mode="asym;dc"

Label the nodes where you want to deploy ack-qat-deviceplugin:

kubectl label node cn-beijing.172.17.XX.XX "alibabacloud.com/type"="ebmg8i"

Step 3: Verify plugin registration

After installation, wait a few minutes for the pod to start, then confirm that ack-qat-deviceplugin is running in the kube-system namespace.

Query the available QAT VFs on your nodes:

kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{"  "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}'

Expected output:

cn-beijing.172.17.XX.XX
  cpu: 189280m
  ephemeral-storage: 113783349470
  hugepages-1Gi: 0
  hugepages-2Mi: 0
  memory: 1027672932Ki
  pods: 2133
  qat.intel.com/cy2_dc2: 32

The qat.intel.com/cy2_dc2: 32 entry confirms that QAT VFs are available. For an ecs.ebmg8i.48xlarge instance, the total VF count equals twice the setup.vf_per_pf value. With the default value of 16, you get 32 VFs.

Step 4: Deploy an application with QAT acceleration

This example uses BoringSSL and Envoy to show how to offload TLS encryption and decryption to QAT hardware. BoringSSL is an open source cryptographic library and Envoy is a cloud-native gateway commonly used for microservices communication.

Create a self-signed certificate and store it as a Kubernetes Secret:

openssl req -x509 -new -batch -nodes -subj '/CN=localhost' -keyout key.pem -out cert.pem
kubectl create secret tls envoy-tls-secret --cert cert.pem --key key.pem

Create an Envoy ConfigMap that enables QAT-backed TLS via the private_key_providers feature:

Envoy configuration file

apiVersion: v1
kind: ConfigMap
metadata:
  name: boringssl-envoy-config
data:
  envoy-conf.yaml: |
    static_resources:
      listeners:
      - address:
          socket_address:
            address: 0.0.0.0
            port_value: 9000
        filter_chains:
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                  certificate_chain: { "filename": "/etc/envoy/tls/tls.crt" }
                  private_key_provider:
                    provider_name: qat     # Enable QAT by setting provider_name and specifying @type in typed_config.
                    typed_config:
                      "@type": "type.googleapis.com/envoy.extensions.private_key_providers.qat.v3alpha.QatPrivateKeyMethodConfig"
                      poll_delay: 0.002s
                      private_key: { filename: "/etc/envoy/tls/tls.key" }
          filters:
          - name: envoy.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: auto
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains:
                  - "*"
                  routes:
                  - match: { prefix: / }
                    direct_response: { status: 200 }
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  @type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
    admin:
      access_log_path: "/dev/null"
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 9001

Deploy the Envoy application and Service. Mount the Secret and ConfigMap created in the previous steps as volumes. All pod specs in this example use the envoy-accel image from OpenAnolis, which includes Intel QAT acceleration support for TLS and Gzip. A single-process Envoy instance requires one QAT device, so qat.intel.com/cy2_dc2 is set to 1 in the resource limits.

YAML file used to deploy the Envoy application and Service

apiVersion: v1
kind: Service
metadata:
  name: helloenvoy
  labels:
    app: boringssl-envoy
spec:
  type: NodePort
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: https
  selector:
    app: boringssl-envoy

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: boringssl-envoy
  labels:
    app: boringssl-envoy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boringssl-envoy
  template:
    metadata:
      labels:
        app: boringssl-envoy
    spec:
      nodeSelector:
        "alibabacloud.com/type": "ebmg8i"
      containers:
        - name: envoy
          image: registry.openanolis.cn/openanolis/envoy-accel:1.26.2-23  # OpenAnolis envoy-accel image with Intel QAT support for TLS and Gzip.
          securityContext:
            privileged: true  # Required for QAT device access.
          imagePullPolicy: IfNotPresent
          args:
          - --cpuset-threads
          command:
          - envoy
          - -c
          - /etc/envoy/config/envoy-conf.yaml
          resources:
            limits:
              qat.intel.com/cy2_dc2: 1  # One QAT device per single-process Envoy instance.
              cpu: 3
              memory: "2G"
            requests:
              cpu: 3
              memory: "2G"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: tls
              mountPath: /etc/envoy/tls
              readOnly: true
            - name: config
              mountPath: /etc/envoy/config
              readOnly: true
            - name: devfs
              mountPath: /dev  # Mount QAT devices to the container.
      volumes:
        - name: devfs
          hostPath:
            path: /dev
            type: ""
        - name: tls
          secret:
            secretName: envoy-tls-secret
        - name: config
          configMap:
            name: boringssl-envoy-config

Verify that QAT hardware is handling the TLS operations.

On the node running Envoy, record the current QAT firmware request count:

# Log on to the node on which Envoy is deployed. 
cat /sys/kernel/debug/qat_4xxx_0000\:e8\:00.0/fw_counters

Expected output before any traffic:

Expected output

+------------------------------------------------+
| FW Statistics for Qat Device                   |
+------------------------------------------------+
| Firmware Requests [AE  0]:                   0 |
| Firmware Responses[AE  0]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  1]:                   0 |
| Firmware Responses[AE  1]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  2]:                   0 |
| Firmware Responses[AE  2]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  3]:                   0 |
| Firmware Responses[AE  3]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  4]:                   0 |
| Firmware Responses[AE  4]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  5]:                   0 |
| Firmware Responses[AE  5]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  6]:                   0 |
| Firmware Responses[AE  6]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  7]:                   0 |
| Firmware Responses[AE  7]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  8]:                   0 |
| Firmware Responses[AE  8]:                   0 |
+------------------------------------------------+

Send a request to the Service using the certificate:

# Access the Service from the client on which the Secret is created. 
kubectl port-forward svc/helloenvoy 32296:9000
curl --cacert cert.pem https://localhost:32296 -v

Expected output:

Expected output

*   Trying 127.0.0.1:32296...
* Connected to localhost (127.0.0.1) port 32296 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=localhost
*  start date: Apr 22 07:32:56 2024 GMT
*  expire date: May 22 07:32:56 2024 GMT
*  common name: localhost (matched)
*  issuer: CN=localhost
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:32296
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Tue, 23 Apr 2024 06:48:53 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact

Check the firmware counter again on the node:

# Log on to the node on which Envoy is deployed. 
cat /sys/kernel/debug/qat_4xxx_0000\:e8\:00.0/fw_counters

Expected output after traffic:

Expected output

+------------------------------------------------+
| FW Statistics for Qat Device                   |
+------------------------------------------------+
| Firmware Requests [AE  0]:                   1 |
| Firmware Responses[AE  0]:                   1 |
+------------------------------------------------+
| Firmware Requests [AE  1]:                   0 |
| Firmware Responses[AE  1]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  2]:                   0 |
| Firmware Responses[AE  2]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  3]:                   0 |
| Firmware Responses[AE  3]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  4]:                   0 |
| Firmware Responses[AE  4]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  5]:                   0 |
| Firmware Responses[AE  5]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  6]:                   0 |
| Firmware Responses[AE  6]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  7]:                   0 |
| Firmware Responses[AE  7]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  8]:                   0 |
| Firmware Responses[AE  8]:                   0 |
+------------------------------------------------+

The increment in Firmware Requests [AE 0] and Firmware Responses [AE 0] confirms that the TLS handshake was processed by QAT hardware rather than the CPU.