All Products
Search

This Product

    Container Service for Kubernetes:Accelerate data encryption, decryption, compression, and decompression by using QAT

    Document Center

    Container Service for Kubernetes:Accelerate data encryption, decryption, compression, and decompression by using QAT

    Last Updated:Apr 30, 2025

    A QuickAssist Technology (QAT) device is an accelerator that is used to enhance system cryptography capability. QAT is applicable to business applications that require high-performance encryption and decryption capabilities. These business applications include web servers, databases, and other services that require a high volume of secure communication. This ensures data security, reduces CPU pressure, and improves overall system performance. You can deploy ack-qat-deviceplugin in an ACK Pro cluster to accelerate operations such as data encryption, decryption, compression, and decompression in business applications by using QAT devices deployed on eighth-generation Elastic Compute Service (ECS) Bare Metal Instances that use Intel Sapphire Rapids.

    Terms

    Intel QAT

    Intel® QAT is a workload acceleration technology that is integrated with Intel® Xeon® Scalable processors. QAT offloads critical workloads of data compression, decompression, encryption, decryption, and public key data encryption from the CPU cores and accelerates these operations. This improves CPU performance and efficiency and lowers data footprint. For more information, see Intel official documentation.

    ack-qat-deviceplugin

    ack-qat-deviceplugin is developed based on open source intel-accel-plugin-qat of OpenAnolis. You can deploy ack-qat-deviceplugin in an ACK Pro cluster to use QAT devices that are deployed on eighth-generation ECS Bare Metal Instances that use Intel Sapphire Rapids. The device plug-in mechanism of Kubernetes exposes QAT devices as on-demand resources to business applications such as NGINX and Envoy that are running in your cluster. This accelerates operations such as data encryption, decryption, compression, and decompression and saves computing resources.

    Limits

    • Instance type: The ECS Bare Metal Instances are in the ecs.ebmg8i and ecs.ebmc8i instance families. In this example, the ecs.ebmg8i.48xlarge instance type is used.

      Note

      The ecs.ebmg8i and ecs.ebmc8i instance families of ECS Bare Metal Instances are only available in select regions. To verify regional availability and inventory status, see Instance Types Available for Each Region.

    • Operating system: Alibaba Cloud Linux UEFI 3.2104 Security Enhanced.

    • If you have deployed a device plug-in that is used to expose QAT devices of other providers, uninstall the device plug-in to prevent conflicts.

    Prerequisites

    Step 1: Configure nodes by using a script

    Before you use QAT devices for the first time, you must update kernel parameters of relevant nodes and restart the nodes by using the following script:

    Important

    To prevent repeated configurations, we recommend that you create a node pool for the instances in the ecs.ebmg8i and ecs.ebmc8i instance families and add the following script to the user data of the node pool. Do not add the script to the pre-defined custom data of the node pool. Otherwise, nodes in the node pool cannot be added to your cluster. For more information, see Create a node pool. 

    yum install kernel-0:5.10.134-16.1.al8.x86_64 -y
if [ $? -ne 0 ];then
    echo "Error: yum update failed"
fi

yum install kernel-modules -y 
if [ $? -ne 0 ];then
    echo "Error: yum install kernel-modules failed"
fi

kernel_path=$(grubby --default-kernel)
kernel_args="intel_iommu=on iommu=pt"
grubby --update-kernel=$kernel_path --args=$kernel_args

reboot -f

    Step 2: Deploy ack-qat-deviceplugin by using Helm

    1. Run the following command to install ack-qat-deviceplugin: 

      helm install ack-qat-deviceplugin https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/pre/charts-incubator/ack-qat-deviceplugin-0.1.2.tgz \
  --set regionId="cn-beijing" \
  --set setup.vf_per_pf="16" \
  --set setup.enabled_mode="asym;dc"

      Parameter

      Required

      Description

      Default value

      regionId

      No

      The region in which the cluster resides. If you specify this parameter, the chart is pulled over the internal network.

      No default value

      setup.vf_per_pf

      No

      The number of virtual functions (VFs) to be created for each physical function (PF). Valid values: 0 to 16.

      16

      setup.enabled_mode

      No

      The driver mode of QAT devices. Valid values: sym, asym, and dc. For more information, see sysfs-driver-qat.

      • sym: accelerates symmetric encryption and decryption.

      • asym: accelerates asymmetric encryption and decryption.

      • dc: accelerates compression and decompression.

      You can specify up to two driver modes. Separate multiple driver modes with semicolons (;). For example, a value of aymc;dc indicates that QAT devices are used to accelerate asymmetric encryption and decryption, compression, and decompression.

      asym;dc

    2. Run the following command to add the default label to the nodes on which you want to deploy ack-qat-deviceplugin: 

      kubectl label node cn-beijing.172.17.XX.XX "alibabacloud.com/type"="ebmg8i"

    3. After the component is installed, wait a few minutes and check whether the status of the component pod is normal. If ack-qat-deviceplugin in the kube-system namespace of the cluster changes to the running state, the component pod runs as expected. Then, run the following command to query the available VFs on the nodes: 

      kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{range $k,$v:=.status.allocatable}}{{"  "}}{{$k}}{{": "}}{{$v}}{{"\n"}}{{end}}{{end}}'

      Expected output:

      cn-beijing.172.17.XX.XX
  cpu: 189280m
  ephemeral-storage: 113783349470
  hugepages-1Gi: 0
  hugepages-2Mi: 0
  memory: 1027672932Ki
  pods: 2133
  qat.intel.com/cy2_dc2: 32

      • qat.intel.com/cy2_dc2: 32 in the output indicates that the number of VFs is 32. For an instance of the ecs.ebmg8i.48xlarge instance type, the number of VFs is twice the value of the setup.vf_per_pf parameter. In this example, the default value 16 is used for the setup.vf_per_pf parameter.

      • cy{} and dc{} in qat.intel.com/cy{}_dc{} indicate the number of work queues for encryption, decryption, compression, and decompression of a QAT device. cy{} indicates the number of work queues for encryption and decryption and dc{} indicates the number of work queues for compression and decompression For an instance of the ecs.ebmg8i.48xlarge instance type, a QAT device has two work queues for encryption and decryption and two work queues for compression and decompression.

    Step 3: Deploy an application and allocate QAT devices

    In this example, BoringSSL and Envoy are used to show how QAT devices are used to accelerate TLS encryption and decryption. BoringSSL is an open source cryptographic library. Envoy is a general cloud-native gateway used for communication among services in the microservices model.

    1. Run the following commands to create a certificate and use the certificate to create a Secret: 

      openssl req -x509 -new -batch -nodes -subj '/CN=localhost' -keyout key.pem -out cert.pem
kubectl create secret tls envoy-tls-secret --cert cert.pem --key key.pem

    2. Use the following code to create an Envoy configuration file as a ConfigMap. The private_key_providers feature must be enabled for Envoy.

      Envoy configuration file

      apiVersion: v1
kind: ConfigMap
metadata:
  name: boringssl-envoy-config
data:
  envoy-conf.yaml: |
    static_resources:
      listeners:
      - address:
          socket_address:
            address: 0.0.0.0
            port_value: 9000
        filter_chains:
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                  certificate_chain: { "filename": "/etc/envoy/tls/tls.crt" }
                  private_key_provider:
                    provider_name: qat     # Enable QAT by configuring the provider_name parameter and specifying @type in the type_config parameter.  
                    typed_config:
                      "@type": "type.googleapis.com/envoy.extensions.private_key_providers.qat.v3alpha.QatPrivateKeyMethodConfig"
                      poll_delay: 0.002s
                      private_key: { filename: "/etc/envoy/tls/tls.key" }
          filters:
          - name: envoy.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              codec_type: auto
              stat_prefix: ingress_http
              route_config:
                name: local_route
                virtual_hosts:
                - name: backend
                  domains:
                  - "*"
                  routes:
                  - match: { prefix: / }
                    direct_response: { status: 200 }
              http_filters:
              - name: envoy.filters.http.router
                typed_config: 
                  @type\": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
    admin:
      access_log_path: "/dev/null"
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 9001

    3. Deploy the Envoy application and Service and mount the preceding Secret and ConfigMap as volumes to the application.

      You must use the envoy-accel image of OpenAnolis to deploy the Envoy application and Service. This image supports Intel QAT acceleration for features such as TLS and GNU zip (Gzip).

      YAML file used to deploy the Envoy application and Service

      apiVersion: v1
kind: Service
metadata:
  name: helloenvoy
  labels:
    app: boringssl-envoy
spec:
  type: NodePort
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: https
  selector:
    app: boringssl-envoy

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: boringssl-envoy
  labels:
    app: boringssl-envoy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boringssl-envoy
  template:
    metadata:
      labels:
        app: boringssl-envoy
    spec:
      nodeSelector:
        "alibabacloud.com/type": "ebmg8i"
      containers:
        - name: envoy
          image: registry.openanolis.cn/openanolis/envoy-accel:1.26.2-23  # Use the envoy-accel image of OpenAnolis. This image supports Intel QAT acceleration for features such as TLS and Gzip. 
          securityContext:
            privileged: true # Specify that privileged containers are created. 
          imagePullPolicy: IfNotPresent
          args:
          - --cpuset-threads
          command:
          - envoy
          - -c
          - /etc/envoy/config/envoy-conf.yaml
          resources:
            limits:
              qat.intel.com/cy2_dc2: 1
              cpu: 3
              memory: "2G"
            requests:
              cpu: 3
              memory: "2G"
          ports:
            - containerPort: 9000
          volumeMounts:
            - name: tls
              mountPath: /etc/envoy/tls
              readOnly: true
            - name: config
              mountPath: /etc/envoy/config
              readOnly: true
            - name: devfs
              mountPath: /dev  # Mount QAT devices to the containers. 
      volumes:
        - name: devfs
          hostPath:
            path: /dev
            type: ""
        - name: tls
          secret:
            secretName: envoy-tls-secret
        - name: config
          configMap:
            name: boringssl-envoy-config

      A single-process Envoy application requires one QAT device for encryption and decryption and one QAT device for compression and decompression. Therefore, the limit for the qat.intel.com/cy2_dc2 resource is set to 1.

    4. Verify that QAT devices are used for encryption and decryption.

      1. Run the following command to record the number of times that QAT devices are used before you perform encryption and decryption: 

        # Log on to the node on which Envoy is deployed. 
cat /sys/kernel/debug/qat_4xxx_0000\:e8\:00.0/fw_counters

        Expected output:

        Expected output

        +------------------------------------------------+
| FW Statistics for Qat Device                   |
+------------------------------------------------+
| Firmware Requests [AE  0]:                   0 |
| Firmware Responses[AE  0]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  1]:                   0 |
| Firmware Responses[AE  1]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  2]:                   0 |
| Firmware Responses[AE  2]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  3]:                   0 |
| Firmware Responses[AE  3]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  4]:                   0 |
| Firmware Responses[AE  4]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  5]:                   0 |
| Firmware Responses[AE  5]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  6]:                   0 |
| Firmware Responses[AE  6]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  7]:                   0 |
| Firmware Responses[AE  7]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  8]:                   0 |
| Firmware Responses[AE  8]:                   0 |
+------------------------------------------------+

      2. Run the following commands to access the Service by using the certificate: 

        # Access the Service from the client on which the Secret is created. 
kubectl port-forward svc/helloenvoy 32296:9000
curl --cacert cert.pem https://localhost:32296 -v

        Expected output:

        Expected output

        *   Trying 127.0.0.1:32296...
* Connected to localhost (127.0.0.1) port 32296 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=localhost
*  start date: Apr 22 07:32:56 2024 GMT
*  expire date: May 22 07:32:56 2024 GMT
*  common name: localhost (matched)
*  issuer: CN=localhost
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: localhost:32296
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< date: Tue, 23 Apr 2024 06:48:53 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact

      3. Run the following command to record the number of times that QAT devices are used after the certificate-based access: 

        # Log on to the node on which Envoy is deployed. 
cat /sys/kernel/debug/qat_4xxx_0000\:e8\:00.0/fw_counters

        Expected output:

        Expected output

        +------------------------------------------------+
| FW Statistics for Qat Device                   |
+------------------------------------------------+
| Firmware Requests [AE  0]:                   1 |
| Firmware Responses[AE  0]:                   1 |
+------------------------------------------------+
| Firmware Requests [AE  1]:                   0 |
| Firmware Responses[AE  1]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  2]:                   0 |
| Firmware Responses[AE  2]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  3]:                   0 |
| Firmware Responses[AE  3]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  4]:                   0 |
| Firmware Responses[AE  4]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  5]:                   0 |
| Firmware Responses[AE  5]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  6]:                   0 |
| Firmware Responses[AE  6]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  7]:                   0 |
| Firmware Responses[AE  7]:                   0 |
+------------------------------------------------+
| Firmware Requests [AE  8]:                   0 |
| Firmware Responses[AE  8]:                   0 |
+------------------------------------------------+

        The output indicates that the number of times that QAT devices are used increases.