Enable KMS secret encryption for an ACK Edge cluster - Container Service for Kubernetes

Prerequisites

Before you begin, make sure you have:

A KMS key in the same region as your cluster. ACK Edge Pro clusters support default keys, software-protected keys, and hardware-protected keys. For more information, see Getting started with Key Management and Billing.
Outbound network access to 100.64.0.0/10. When Secret encryption is enabled, the control plane calls the KMS API to encrypt and decrypt Secrets. The outbound rules of your security group or VPC network ACL must allow traffic to the CIDR block 100.64.0.0/10. For more information, see Configure security groups for clusters.
The required authorization, based on your account type:
- Alibaba Cloud account: The account must be authorized to assume the AliyunCSManagedSecurityRole role. If it isn't, the ACK console prompts you to authorize during enablement. Alternatively, go to the RAM Quick Authorization page and complete the authorization.
- RAM user or RAM role: Must have administrator or O&M engineer role-based access control (RBAC) permissions on the cluster (see Grant RBAC permissions to a RAM user or RAM role) and must be granted the AliyunKMSCryptoAdminAccess permission (see Grant permissions to a RAM user or RAM role).

Limitations

Do not delete or disable the encryption key. After Secret encryption is enabled, never use the KMS API or KMS console to disable or delete the key. If the key is deleted, the API server cannot decrypt Secrets or service account objects, which causes service interruptions.
Region constraint. The KMS key must belong to the same region as the ACK Edge Pro cluster.

How it works

Kubernetes Secrets are used to store and manage sensitive data, such as application passwords, Transport Layer Security (TLS) certificates, and credentials for downloading Docker images. ACK Edge Pro clusters use the KMS provider mechanism of Kubernetes, which implements envelope encryption for Secrets stored in etcd.

Encryption flow:

When a Secret is stored, the API server generates a random data encryption key (DEK) and uses it to encrypt the Secret.
The API server sends the DEK to KMS. KMS encrypts the DEK using the specified KMS key and returns the encrypted DEK.
The API server stores both the encrypted Secret and the encrypted DEK in etcd.

Decryption flow:

When a Secret is read, the API server sends the encrypted DEK to KMS, which decrypts it using the KMS key.
The API server uses the plaintext DEK to decrypt the Secret and returns the plaintext Secret.

For background on the underlying mechanisms, see The KMS provider and Use envelope encryption.

Enable Secret encryption for a new cluster

Log on to the ACK console. In the left navigation pane, click Clusters.
In the upper-right corner of the Clusters page, click Create Kubernetes Cluster.
On the Create Cluster page, click the ACK Edge tab. Scroll to the lower part of the tab and click Advanced Options (Optional). Find the Secret Encryption parameter, click Select Key, and then select a key ID from the drop-down list. If you haven't created a KMS key yet, click create keys to open the KMS console and create one. For more information, see Create a CMK. For other cluster configuration options, see Create an ACK Edge cluster.
Verify that Secret encryption is active by checking ActionTrail. Log on to the ActionTrail console. In the left-side navigation pane, choose Events > Event Query. On the Event Query page, confirm that the aliyuncsmanagedsecurityrole system role appears in encryption and decryption event logs.

Enable Secret encryption for an existing cluster

Important

After you enable Secret encryption, do not use the KMS API or the KMS console to disable or delete the key that is used to encrypt and decrypt Secrets. Otherwise, the API server becomes unavailable and cannot retrieve Secrets or service account objects, causing service interruptions.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your ACK Edge Pro cluster. On the cluster details page, click the Basic Information tab. In the Basic Information section, turn on Secret Encryption. If this is the first time you enable Secret encryption, follow the on-screen instructions and click Go to RAM console to open the Cloud Resource Access Authorization page. Click Confirm Authorization Policy.

To enable Secret encryption, the RAM user or RAM role you use must have the administrator or O&M engineer RBAC role on the cluster. To assign the AliyunCSManagedSecurityRole role, log on with an Alibaba Cloud account or a RAM user or RAM role that has RAM management permissions.
In the Secret Encryption dialog box, select an existing key and click OK. If no key is available, click create keys to create one in the KMS console. For more information, see Create a CMK.
Wait for the cluster status to change from Updating to Running. Secret encryption is now active.

To disable Secret encryption, turn off Secret Encryption in the Basic Information section.

Rotate encryption keys

KMS supports automatic key rotation. During a rotation, existing Secrets remain encrypted with the original key. New Secrets are encrypted with the new key.

To force all existing Secrets to be re-encrypted with the new key after a rotation, run:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

For more information about setting up automatic key rotation, see Configure key rotation.

FAQ

Does kubectl return ciphertext after I enable Secret encryption?

No. kubectl always returns plaintext when querying a Secret. Secret encryption operates at the etcd layer — Secrets are stored as ciphertext in etcd, but the API server decrypts them before returning the result to kubectl.

Are existing Secrets automatically re-encrypted after a key rotation?

No. During a key rotation, existing Secrets remain encrypted with the original key version. Only new Secrets use the new key. To re-encrypt all existing Secrets with the new key, run the command in Rotate encryption keys.

How do I prevent RAM users or RAM roles from enabling or disabling Secret encryption?

Attach the following deny policy to the RAM user or RAM role. For more information, see Grant access to clusters and cloud resources using RAM.

{
    "Action": [
        "cs:UpdateKMSEncryption"
    ],
    "Effect": "Deny",
    "Resource": [
        "*"
    ]
}