Enable KMS encryption for Secrets at rest in ACK Pro clusters - Container Service for Kubernetes

Prerequisites

Before you begin, ensure that you have:

A KMS key in the same region as your ACK Pro cluster. ACK Pro clusters support default keys, software-protected keys, and hardware-protected keys. For details, see Getting started with keys and Billing.
Authorization for the account you use to enable Secret encryption:
- Alibaba Cloud account: must be authorized to assume the AliyunCSManagedSecurityRole role. If not yet authorized, the ACK console will prompt you to authorize when you enable Secret encryption. Follow the console instructions, or go directly to the Cloud Resource Access Authorization page.
- RAM user or RAM role: must have both of the following:
  - Administrator or O&M engineer RBAC permissions on the ACK Pro cluster. See Grant RBAC permissions to RAM users or RAM roles.
  - The AliyunKMSCryptoAdminAccess permission. See Grant permissions to a RAM user or RAM role.

Limitations

Secret encryption is supported only on ACK Pro clusters and ACK Serverless Pro clusters.
After enabling Secret encryption, do not use the KMS API or KMS console to disable or delete the key. If the key becomes unavailable, the API server cannot retrieve Secrets or service account objects, causing service interruptions.
To disable Secret encryption, turn off Secret Encryption in the cluster's Basic Information tab.

How it works

ACK uses the Kubernetes KMS provider mechanism with envelope encryption to protect Secrets in etcd. Envelope encryption adds two layers of protection: the Secret is encrypted with a data encryption key (DEK), and the DEK itself is encrypted with a key encryption key (KEK) stored in KMS. This means the DEK is never stored in plaintext alongside the data it protects.

Encryption:

When you store a Kubernetes Secret, the API server generates a random DEK and uses it to encrypt the Secret.
The API server sends the DEK to KMS. KMS encrypts the DEK with your specified KMS key (the KEK) and returns the encrypted DEK.
The API server stores both the encrypted Secret and the encrypted DEK in etcd.

Decryption:

The API server calls the KMS Decrypt operation to decrypt the DEK.
The API server uses the plaintext DEK to decrypt the Secret and returns it to the caller.

For more information, see The KMS provider and Use envelope encryption.

Enable encryption at rest for Secrets

New ACK Pro cluster

Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click Cluster Templates in the upper-right corner. On the Select Cluster Template page, find Professional Managed Kubernetes Cluster and click Create.
On the Managed Kubernetes tab, click Show Advanced Options at the bottom of the page. In the Secret Encryption section, select Select Key and choose a key from the drop-down list. Set the other parameters based on your requirements and click Create Cluster. For details, see Create an ACK Pro cluster.

Warning

Do not disable or delete the KMS key after enabling Secret encryption. If the key becomes unavailable, the API server cannot retrieve Secrets or service account objects, causing service interruptions.

Verify: Log on to the ActionTrail console. In the left-side navigation pane, click Event Detail Query. Check for encryption and decryption operations performed by assuming the aliyuncsmanagedsecurityrole role. If these operations appear, Secret encryption is active.

Existing ACK Pro cluster

Log on to the ACK console and click Clusters in the left-side navigation pane.
On the Clusters page, click the name of the cluster you want to manage. On the cluster details page, click the Basic Information tab. In the Basic Information section, turn on Secret Encryption. If this is the first time you enable Secret encryption, click Go to RAM console when prompted to navigate to the Cloud Resource Access Authorization page, then click Confirm Authorization Policy.

To enable Secret encryption, the RAM user or RAM role you use must have the administrator or O&M engineer RBAC role on the cluster. To assign the AliyunCSManagedSecurityRole, log on with an Alibaba Cloud account or a RAM user or RAM role that has RAM management permissions.
In the Secret Encryption dialog box, select a key and click OK. If no key is available, click create keys to create one in the KMS console. For details, see Create a CMK. When the cluster status changes from Updating to Running, Secret encryption is enabled.

Warning

Do not disable or delete the KMS key after enabling Secret encryption. If the key becomes unavailable, the API server cannot retrieve Secrets or service account objects, causing service interruptions.

To disable: In the Basic Information section of the cluster details page, turn off Secret Encryption.

Rotate the encryption key

KMS automatic key rotation encrypts new Secrets with the rotated key. Existing Secrets remain encrypted with the original key until you explicitly re-encrypt them.

Manual re-encryption: To force all existing Secrets to be re-encrypted with the new key, run the following command after rotation completes:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

For details on configuring automatic key rotation in KMS, see Configure key rotation.

FAQ

Does kubectl return ciphertext after Secret encryption is enabled?

No. Secret encryption protects Secrets at the etcd storage layer — Secrets are stored as ciphertext in etcd. When you query a Secret through kubectl, the API server decrypts it before returning the result, so kubectl always returns plaintext.

How do I prevent RAM users or RAM roles from enabling or disabling Secret encryption for existing ACK Serverless Pro clusters?

Attach a deny policy for the cs:UpdateKMSEncryption action to the RAM users or RAM roles. For details on creating custom policies, see Create a custom RAM policy.

{
    "Action": [
        "cs:UpdateKMSEncryption"
    ],
    "Effect": "Deny",
    "Resource": [
        "*"
    ]
}