Inclavare Containers implements Enclave Attestation Architecture (EAA), which is a universal and cross-platform infrastructure that is used for remote attestation. EAA can prove that sensitive workloads are running in a genuine and hardware-based Trusted Execution Environment (TEE). This topic describes how to use confidential containers to implement remote attestation in a managed Kubernetes cluster for confidential computing.

Prerequisites

Background information

EAA uses a TLS certificate in which a quote of a hardware-based TEE is embedded. This ensures that the server and the client communicate in a hardware-based TEE. The following figure shows the workflow and architecture of EAA.

AKK

If you want to verify that your workloads are running on a trusted platform, you can start Shelter and send the request to Inclavared for verification. The following procedure shows how to verify that your workloads are running on a trusted platform:

  1. After Inclavared receives the request from Shelter, Inclavared sends the request to a confidential container. Both Inclavared and the confidential container generate a TLS certificate in which a quote of a hardware-based TEE is embedded.
  2. An attested and secure channel is established between Inclavared and Shelter based on Enclave-TLS.
  3. A mutually attested and secure channel is established between Inclavared and the confidential container based on Enclave-TLS.
  4. Inclavared forwards the information about the hardware-based TEE and the sensitive information from the confidential container to Shelter.
  5. Shelter verifies the measurements of the enclave runtime and returns the result.

The following table describes the components of EAA.

Component Role Description
Confidential container Workload Runs the server program enclave-tls-server in Occlum. The confidential container also responds to the request from Inclavared based on Enclave-TLS and returns the attestation evidence of the confidential container. The attestation evidence contains the values of mrenclave and mrsigner. For more information, see Occlum and enclave-tls.
Inclavared Attester Forwards the traffic between the downstream confidential container and the upstream Shelter client. The communication is protected by the attested Enclave-TLS channel.
Shelter On-premises verifier
  1. Records the launch measurements of the enclave runtime.
  2. Establishes the attested Enclave-TLS channel to communicate with Inclavared.
  3. Shelter verifies the launch measurements of the enclave runtime. This way, you can verify that your workloads are running in a genuine hardware-based TEE.
Alibaba Cloud Provisioning Certificate Caching Service (PCCS) Remote attestation service The Alibaba Cloud SGX remote attestation service is fully compatible with the remote attestation service for Intel® SGX Elliptic Curve Digital Signature Algorithm (ECDSA) and the Intel® SGX SDK. Therefore, the vSGX instances of Alibaba Cloud, which are instances that consist of the g7t, c7t, and r7t instance types, can gain trust from remote providers and producers by using remote attestation. For more information, see Attestation Services for Intel® SGX ECDSA and the Intel® SGX SDK.

Limits

  • Remote attestation can be implemented only on applications that are created by using DaemonSets. This means that an application must have one workload on each node in the cluster.
  • To enable direct communication between Shelter and the workload, each node in the cluster must be associated with an elastic IP address (EIP).

Step 1: Deploy Inclavared

  1. Run the following command to create an application named Inclavared by using the Daemonset:
    cat <<-EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: inclavared
    spec:
      selector:
        matchLabels:
          k8s-app: inclavared
      template:
        metadata:
          labels:
            k8s-app: inclavared
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: alibabacloud.com/tee-hardware-category
                    operator: In
                    values:
                    - intel-sgx1
                    - intel-sgx2
          containers:
          - image: docker.io/inclavarecontainers/inclavared:latest
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            name: inclavared
            command:
              - /usr/local/bin/inclavared
            args:
              - --listen
              - 0.0.0.0:1236
              - --xfer
              - 127.0.0.1:1234
              - --attester
              - sgx_ecdsa
              - --verifier 
              - sgx_ecdsa_qve
              - --mutual
            volumeMounts:
            - mountPath: /dev/sgx/enclave
              name: dev-enclave
            - mountPath: /dev/sgx/provision
              name: dev-provision
            - mountPath: /var/run/aesmd
              name: run-dir
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
              limits:
                cpu: 1
                memory: 1000Mi
          tolerations:
          - effect: NoSchedule
            key: alibabacloud.com/sgx_epc_MiB
            operator: Exists
          volumes:
          - hostPath:
              path: /var/run/aesmd
              type: DirectoryOrCreate
            name: run-dir
          - hostPath:
              path: /dev/sgx_enclave
            name: dev-enclave
          - hostPath:
              path: /dev/sgx_provision
            name: dev-provision
          hostNetwork: true
    EOF
  2. Run the following command to check whether the Inclavared application is deployed:
    kubectl get daemonset inclavared

    Expected output:

    NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    inclavared   2         2         2       2            2           <none>          7d22h
    Note After the Inclavared application is started:
    1. The listen address is set to 0.0.0.0:1236 for the Inclavared application to listen for requests from Shelter.
    2. After the Inclavared application receives a request from Shelter, the Inclavared application sends the request to the workload on the current node.

Step 2: Deploy the occlum-attestation-app workload

  1. Run the following command to create the workload named occlum-attestation-app:
    cat <<-EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: occlum-attestation-app
    spec:
      selector:
        matchLabels:
          k8s-app: occlum-attestation-app
      template:
        metadata:
          labels:
            k8s-app: occlum-attestation-app
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: alibabacloud.com/tee-hardware-category
                    operator: In
                    values:
                    - intel-sgx1
                    - intel-sgx2
          containers:
          - image: docker.io/inclavarecontainers/occlum-ecdsa-server:0.21.0
            imagePullPolicy: IfNotPresent
            securityContext:
              privileged: true
            name: occlum-attestation-app
            command:
              - /bin/enclave-tls-server
            args:
              - --ip
              - "127.0.0.1"    #The IP address of your on-premises machine. 
              - --port
              - "1234"
              - --mutual
            env:
              - name: ENCLAVE_TYPE
                value: intelSgx
              - name: RUNE_CARRIER
                value: occlum
              - name: ENCLAVE_RUNTIME_LOGLEVEL
                value: info
              - name: ENCLAVE_RUNTIME_PATH
                value: /opt/occlum/build/lib/libocclum-pal.so.0.21.0
              - name: ENCLAVE_RUNTIME_ARGS
                value: occlum_workspace_server
              - name: OCCLUM_RELEASE_ENCLAVE
                value: "1"
            workingDir: /run/rune
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
              limits:
                cpu: 1
                memory: 1000Mi
          tolerations:
          - effect: NoSchedule
            key: alibabacloud.com/sgx_epc_MiB
            operator: Exists
          hostNetwork: true
    EOF
  2. Run the following command to check whether the occlum-attestation-app workload is deployed:
    kubectl get daemonset occlum-attestation-app

    Expected output:

    NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    occlum-attestation-app   2         2         2       2            2           <none>          7d22h

    The preceding output indicates that the occlum-attestation-app workload is deployed. The enclave-tls-server program in the workload listens for requests from Inclavared at the listen address 127.0.0.1:1234.

Step 3: Modify the security group settings of the cluster

  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. On the details page of the cluster, click the Cluster Resources tab and click the hyperlink on the right side of Security Group.
  5. Modify the default security group of the cluster and allow inbound traffic on port 1236. For more information, see Modify security group rules.

Step 4: Associate an EIP with an ECS instance

Note If the ECS instance is associated with an EIP, skip this step.
  1. Log on to the ACK console.
  2. In the left-side navigation pane of the ACK console, click Clusters.
  3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.
  4. In the left-side navigation pane of the details page, choose Nodes > Nodes.
  5. On the Nodes page, find the node that you want to manage and click the Instance ID of the node.
  6. In the ECS console, choose Instances & Images > Instances. On the page that appears, click the Instance Details tab.
  7. Click Bind EIP and associate an EIP with the ECS instance.
    akk1
    • If you have created EIPs, select an EIP from the EIP drop-down list in the Bind EIP dialog box.
    • If no EIPs are available, click Create EIP in the Bind EIP dialog box. For more information, see Apply for EIPs. jm11

Step 5: Install Shelter for remote attestation

  1. Install the SGX Platform Software (PSW).
    Note Shelter must rely on the dynamic library provided by the SGX PSW to verify the TLS certificate in which information about the SGX is embedded. For more information, see SGX PSW.
    • On CentOS 8.2, run the following command to install the SGX PSW:
      yum install -y yum-utils && \
          wget -c https://download.01.org/intel-sgx/sgx-linux/2.13/distro/centos8.2-server/sgx_rpm_local_repo.tgz && \
          tar xzf sgx_rpm_local_repo.tgz && \
          yum-config-manager --add-repo sgx_rpm_local_repo && \
          yum makecache && rm -f sgx_rpm_local_repo.tgz && \
          yum install --nogpgcheck -y libsgx-dcap-quote-verify \
          libsgx-dcap-default-qpl libsgx-dcap-ql \
          libsgx-uae-service
    • On Ubuntu 18.04, run the following command to install the SGX PSW:
      apt-get update -y && apt-get install -y wget gnupg && \
          echo "deb [arch=amd64] https://download.01.org/intel-sgx/sgx_repo/ubuntu bionic main" | tee /etc/apt/sources.list.d/intel-sgx.list && \
          wget -qO - https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key | apt-key add - && \
          apt-get update -y && \
          apt-get install -y libsgx-dcap-quote-verify=1.10.100.4-bionic1 \
          libsgx-dcap-default-qpl=1.10.100.4-bionic1 \
          libsgx-dcap-ql=1.10.103.1-bionic1 \
          libsgx-uae-service=2.13.100.4-bionic1
  2. Configure the public endpoint of Alibaba Cloud PCCS.

    The Alibaba Cloud SGX remote attestation service is regionally deployed. For optimal stability, you can access this service in the region where the vSGX instance is deployed. You must manually modify the /etc/sgx_default_qcnl.conf file to adapt to the Alibaba Cloud SGX remote attestation service that is deployed in the region where the vSGX instance is deployed.

    Note Only regions in mainland China support the Alibaba Cloud SGX remote attestation service. For more information, see Regions and zones.
    • If a public IP address is assigned to the vSGX instance, modify /etc/sgx_default_qcnl.conf to the following content:
      # PCCS server address
      PCCS_URL=https://sgx-dcap-server.<Region-ID>.aliyuncs.com/sgx/certification/v3/
      # To accept insecure HTTPS cert, set this option to FALSE
      USE_SECURE_CERT=TRUE
    • If the vSGX instance is in a virtual private cloud (VPC) and has only an internal IP address, modify /etc/sgx_default_qcnl.conf to the following content:
      # PCCS server address
      PCCS_URL=https://sgx-dcap-server-vpc.<Region-ID>.aliyuncs.com/sgx/certification/v3/
      # To accept insecure HTTPS cert, set this option to FALSE
      USE_SECURE_CERT=TRUE
      Note You must replace <Region-ID> with the ID of the region where the vSGX instance is deployed.
  3. Install Shelter for remote attestation.
    • On CentOS 8.2, run the following command to install Shelter:
      yum-config-manager --add-repo https://mirrors.openanolis.org/inclavare-containers/rpm-repo/ && \
          rpm --import https://mirrors.openanolis.org/inclavare-containers/rpm-repo/RPM-GPG-KEY-rpm-sign && \
          yum install -y shelter
    • On Ubuntu 18.04, run the following command to install Shelter:
      echo 'deb [arch=amd64] https://mirrors.openanolis.org/inclavare-containers/deb-repo bionic main' | tee /etc/apt/sources.list.d/inclavare-containers.list && \
          wget -qO - https://mirrors.openanolis.org/inclavare-containers/deb-repo/DEB-GPG-KEY.key  |  apt-key add - && \
          apt-get update -y && apt-get install -y shelter
    Verify the result

    Run the following command to check whether Shelter is installed:

    which shelter

    Expected output:

    /usr/local/bin/shelter

Verify that remote attestation is implemented by using the confidential container in the managed Kubernetes cluster for confidential computing.

Run the following command on each node in the cluster to launch Shelter:

Note Before you run the command, replace <$IP> with the EIP of the worker node.
shelter remoteattestation --verifier sgx_ecdsa --tls openssl --crypto openssl --addr=tcp://<$IP>:1236
  • If Shelter is running in an SGX environment, the following output is returned: jm67

    If remote attestation is implemented, Remote attestation is successful appears in the output.

  • If Shelter is running in a non-SGX environment, the following errors are returned together with the Remote attestation is successful message:
    [load_qve ../sgx_dcap_quoteverify.cpp:209] Error, call sgx_create_enclave for QvE fail [load_qve], SGXError:2006.
    [sgx_qv_get_quote_supplemental_data_size ../sgx_dcap_quoteverify.cpp:527] Error, failed to load QvE.

    The preceding errors have no impact on the remote attestation result. You can ignore the errors.

    The errors are returned because Shelter attempts to load Quote Verification Enclave (QVE) when Shelter calls sgx_qv_get_quote_supplemental_data_size(). If QVE fails to be loaded, Shelter returns the preceding errors and uses Quote Verification Library (QVL) for subsequent attestation. QVE cannot be loaded in non-SGX environments. In non-SGX environments, the attestation is handled by QVL. For more information, see Attestation services for Intel® SGX ECDSA.