Encrypt Kubernetes Secrets at rest with KMS | ACS - Container Compute Service

In Container Compute Service (ACS), you can use a key created in Alibaba Cloud Key Management Service (KMS) to encrypt Kubernetes Secrets at rest. Use a KMS-managed key to protect Secret data stored in the cluster's etcd.

Prerequisites

A KMS key is created in the KMS console. The key must be in the same region as the target ACS cluster.
Important
Secret encryption at rest requires ACS cluster version 1.28 or later.
You have the required permissions for your account type.
- Alibaba Cloud account: The Container Service account is authorized to assume the AliyunCCManagedSecurityRole role. If not yet authorized, the ACS console prompts you when you enable Secret encryption at rest. You can also authorize on the Cloud Resource Access Authorization page.
- RAM user or RAM role:
  - You have the administrator or O&M permissions on the cluster based on RBAC. Configure RBAC permissions for a RAM user or RAM role.
  - You have the AliyunKMSCryptoAdminAccess system permission. Grant RAM permissions to a RAM user or RAM role.

How Secret encryption works

Kubernetes Secrets store sensitive data such as passwords, TLS certificates, and image pull credentials. All Secret data is persisted in the cluster's etcd.

ACS clusters use the community KMS v2 API for encryption at rest, supporting higher Secret capacity and better performance. ACS supports default, software-protected, and hardware-protected keys. Get started with key management. Billing.

Important

After you enable encryption at rest, do not disable or delete the key that is used for Secret encryption and decryption in the KMS console or by calling an API operation. Otherwise, the cluster API server becomes unavailable and you can no longer retrieve objects such as Secrets and ServiceAccounts. This disrupts running applications.

Using a KMS provider for data encryption. Use KMS keys for envelope encryption.

Enable at-rest encryption for ACS clusters

During cluster creation

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click Create Kubernetes Cluster in the upper-left corner.
At the bottom of the page, expand Show Advanced Options, find Secret Encryption, select the Select Key checkbox, and then select a KMS key ID from the drop-down list. Configure other parameters as required and click Confirm. Create an ACS cluster.

Log on to the ActionTrail console and click Event Query in the left-side navigation pane. If the Event Query page shows encryption and decryption event logs generated by the aliyuncsmanagedsecurityrole system role, Secret encryption at rest is enabled.

To disable Secret encryption at rest, go to the Cluster Information > Basic Information tab in the Container Service for Kubernetes (ACK) console, and then turn off the Secret Encryption switch in the Security and Auditing section.

For an existing cluster

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Cluster Information.
On the cluster details page, click the Basic Information tab. In the Security and Auditing section, click Enable next to Secret Encryption.
In the Secret Encryption dialog box, select an existing KMS key and click OK.
If you have not created a KMS key, click Create Key to go to the KMS console to create one. Create a key.
When the cluster status changes from Updating to Running, Secret encryption at rest is enabled.
To disable encryption at rest for Secrets, turn off the Secret Encryption switch in the Security and Auditing section.

Automatic key rotation

KMS supports automatic key rotation for Secret encryption at rest. After rotation, existing Secrets remain encrypted with the previous key version, and new Secrets are encrypted with the new key version. Key rotation.

To ensure that existing Secrets are also encrypted with the new key version, run the following command to forcibly re-encrypt all existing Secrets with the new key version after the key is automatically rotated.

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

FAQ

Will `kubectl get secrets` return ciphertext?

No. The encryption at rest feature encrypts the raw data stored in etcd. This means that after you enable encryption at rest for Secrets, the Secret data stored in etcd is ciphertext. However, the Secret data a client obtains through the Secrets API is still plaintext.

How do I restrict who can manage Secret encryption?

To prevent a RAM user or RAM role from enabling or disabling Secret encryption at rest in an existing cluster, attach the following Deny policy. Grant RAM permissions to a RAM user or RAM role.

  {
      "Action": [
          "cs:UpdateKMSEncryption"
      ],
      "Effect": "Deny",
      "Resource": [
          "*"
      ]
  }