In Container Service for Kubernetes (ACK) Pro clusters, you can use keys that are created in Key Management Service (KMS) to encrypt Kubernetes Secrets. This topic describes how to use a key that is managed by KMS to encrypt Secrets for an ACK Pro cluster.

Table of contents

Prerequisites

  • A customer master key (CMK) is created in the KMS console. For more information, see Create a CMK.
    Note ACK Pro clusters support only CMKs of the Aliyun_AES_256 type for Secret encryption.
  • The required permissions are granted to the Alibaba Cloud account, Resource Access Management (RAM) user, or RAM role that you want to use.
    • Your Alibaba Cloud account can assume the AliyunCSManagedSecurityRole role. Otherwise, the system prompts you to perform the authorization when you enable Secret encryption.
    • If you log on to the console as a RAM user or RAM role, make sure that the RAM user or RAM role has the permissions to access KMS resources, the cs:UpdateKMSEncryption permission, and the predefined role-based access control (RBAC) permissions of ACK. This way, you can enable or disable Secret encryption for ACK Pro clusters.
      View the procedure for granting permissions to a RAM user or RAM role
      • Grant the AliyunKMSCryptoAdminAccess permission. For more information, see Grant permissions to a RAM user or RAM role.
      • Grant the cs:UpdateKMSEncryption permission. The following section describes the policy. For more information, see Create a custom RAM policy. 
          {
      "Action": [
          "cs:UpdateKMSEncryption"
      ],
      "Effect": "Allow",
      "Resource": [
          "*"
      ]
  }
      • Grant the RBAC permissions that are predefined by the administrator role or the O&M engineer role of ACK. The permissions take effect on all namespaces of the specified cluster. For more information, see Grant RBAC permissions to RAM users or RAM roles.
  • The balance in your Alibaba Cloud account is sufficient.

    You are charged by KMS for key management and API calls (on a per 10,000 calls basis). After Secret encryption is enabled for an ACK Pro cluster, kube-apiserver must call the encryption and decryption API operations of KMS to read and write Secrets. By default, a large number of API calls are made to read and write Secrets when you manage the lifecycle of service accounts in the cluster. If the cluster contains a large number of service accounts or Secrets, the cost of KMS API calls is high. Make sure that your account balance is sufficient. Otherwise, service interruptions may occur. If you are not familiar with the pricing rules or your account balance is insufficient, you can disable Secret encryption for the cluster. For more information, see Enable Secret encryption for an ACK Pro cluster. If your account has an overdue payment for more than seven days, you cannot manage the cluster. For more information about the billing rules for KMS, see Billing of KMS.

Limits and usage notes

After you enable Secret encryption, do not use the KMS API or the KMS console to disable or delete the CMK that is used to encrypt and decrypt Secrets, or create a schedule to delete the CMK. Otherwise, the API server becomes unavailable and cannot retrieve Secrets or ServiceAccount objects. As a result, service interruptions occur.

Overview of Secret encryption

Kubernetes Secrets are used to store and manage sensitive data, such as passwords of applications, Transport Layer Security (TLS) certificates, and credentials to download Docker images. Kubernetes stores Secrets in the etcd of a cluster. For more information, see Secrets.

ACK Pro clusters allow you to use a key created in KMS to encrypt Secrets. The KMS provider mechanism of Kubernetes is used during encryption. A KMS provider uses envelope encryption to encrypt and decrypt Secrets that are stored in etcd. Procedures for Secret encryption and decryption:
  • When you use a Kubernetes Secret to store a password, the API server generates a random data encryption key (DEK) to encrypt the Secret. Then, the API server sends the DEK to KMS. KMS uses the specified key to encrypt the DEK and returns the encrypted DEK to the API server. The API server then stores the encrypted Secret and DEK in etcd.
  • When you decrypt the Kubernetes Secret, the system calls the Decrypt operation of KMS to decrypt the data key first. Then, the system uses the plaintext of the data key to decrypt the Kubernetes Secret and returns the decrypted Secret.

For more information, see The KMS provider and Use envelope encryption to encrypt and decrypt local data.

Enable Secret encryption for an ACK Pro cluster

Enable Secret encryption when you create an ACK Pro cluster

  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click Cluster Templates in the upper-right corner. On the Select Cluster Template page, find Professional Managed Kubernetes Cluster and click Create.
  3. On the Managed Kubernetes tab, click Show Advanced Options at the bottom of the page. Find Secret Encryption, select Select Key, and then select a key from the drop-down list. Set the other parameters based on your business requirements and click Create Cluster. For more information about how to configure an ACK Pro cluster, see Create an ACK Pro cluster. Secret encryption

Log on to the ActionTrail console. In the left-side navigation pane, click Event Detail Query. On the Event Detail Query page, check for encryption and decryption operations that are performed by assuming the aliyuncsmanagedsecurityrole role. If these operations exist, the Secret encryption feature is enabled.

Note If you no longer need to use the Secret encryption feature, click the cluster name in the cluster list. On the Cluster Information page, click the Basic Information tab. In the Basic Information section, disable Secret Encryption.

Enable Secret encryption for an existing ACK Pro cluster

  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click the name of the ACK Pro cluster that you want to manage. On the details page of the cluster, click the Basic Information tab. In the Basic Information section, turn on Secret Encryption.

    If this is the first time that you enable Secret encryption, follow the instructions and click Go to RAM console to navigate to the Cloud Resource Access Authorization page. Then, click Confirm Authorization Policy.

    Note
    • If you want to enable Secret encryption, make sure that the RAM user or RAM role that you use is assigned one of the following RBAC roles: the administrator role and the O&M engineer role. For more information, see Grant RBAC permissions to RAM users or RAM roles.
    • If you want to assign the aliyuncsmanagedsecurityrole role, make sure that you log on to the ACK console with an Alibaba Cloud account or a RAM user or RAM role that has the RAM management permissions.
  3. In the Secret Encryption dialog box, select an existing key and click OK.

    If no key is available, click create keys to create a key in the KMS console. For more information, see Create a CMK.

    If the status of the cluster changes from Updating to Running, the Secret encryption feature is enabled for the cluster.

Note If you no longer need the Secret encryption feature, you can turn off Secret Encryption in the Basic Information section.

Use automatic key rotation to encrypt Secrets

You can use the automatic key rotation feature provided by KMS to encrypt Secrets. During a key rotation, the system still uses the original key to encrypt existing Secrets. New Secrets are encrypted by using the new key. For more information about automatic key rotation, see Automatic key rotation.

To force the system to use the new key to encrypt existing Secrets, run the following command after the key is rotated: 

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

FAQ

After Secret encryption is enabled, is ciphertext returned if I use kubectl to query a Secret?

No. After Secret encryption is enabled, plaintext is returned if you use kubectl to query a Secret. The Secret encryption feature encrypts the Secrets that are stored in etcd. After you enable Secret encryption, Secrets are stored in etcd as ciphertext. However, if you use a kubectl client to query a Secret by calling the Secret API provided by the API server of the cluster, plaintext is returned for the Secret.

How do I prohibit RAM users or RAM roles from enabling or disabling the Secret encryption feature for existing ACK Pro clusters?

To prohibit RAM users or RAM roles from enabling or disabling the Secret encryption feature for existing ACK Pro clusters, attach the following policy to the RAM user or RAM role. For more information, see Create a custom RAM policy. 
  {
      "Action": [
          "cs:UpdateKMSEncryption"
      ],
      "Effect": "Deny",
      "Resource": [
          "*"
      ]
  }