In an ACK Pro cluster or , you can use keys from Alibaba Cloud Key Management Service (KMS) to encrypt Kubernetes secrets. This topic describes how to use a KMS-managed key to encrypt Kubernetes secret data at rest in an ACK Pro cluster.
Prerequisites
|
Item |
Description |
|
KMS key |
You have created a KMS key in the Key Management Service console, and the key is in the same region as the target ACK Pro cluster. ACK Pro clusters support default keys, software keys, and hardware keys. You can select a key type. For more information about key management and related operations, see Key management quick start. For details about KMS billing, see billing. Important
After you enable encryption at rest, do not use the KMS console or API to disable or delete the key that is used for secret encryption and decryption. Otherwise, the cluster API server becomes unavailable. As a result, you cannot retrieve objects such as secrets and ServiceAccounts, which disrupts your applications. |
|
Cluster network ACL rules |
When encryption at rest is enabled, the KMS plug-in on the control plane must access Alibaba Cloud KMS APIs to encrypt and decrypt secrets. To allow this, you must add rules that permit traffic to the |
|
Authorization |
Make sure that the required permissions are granted based on your account type.
|
How secret encryption works
In a Kubernetes cluster, sensitive information for applications, such as passwords, TLS certificates, and Docker image credentials, is typically stored and managed by using secret objects. Kubernetes stores all secret object data in the cluster's etcd database. For more information about secrets, see Secrets.
In an ACK Pro cluster or , you can use keys created in KMS to encrypt Kubernetes secrets. The encryption process is based on the KMS Encryption Provider mechanism in Kubernetes, which uses envelope encryption to automatically encrypt and decrypt Kubernetes secrets stored in etcd. The encryption and decryption process is as follows:
-
When you save a secret by using the Kubernetes API, the API server encrypts the secret with a randomly generated data encryption key. Then, the data encryption key is encrypted with your specified KMS key. The encrypted secret and the encrypted data key are stored in etcd.
-
When you retrieve a secret, the system first calls the KMS Decrypt API to decrypt the encrypted data key. Then, it uses the resulting plaintext key to decrypt the secret data and returns the plaintext secret.
For more information, see KMS Encryption Provider mechanism and Use a KMS key for envelope encryption.
KMS v2
Starting from Kubernetes 1.31, ACK Pro clusters and use the community KMS v2 API for encryption at rest by default. KMS v2 supports more secrets in a cluster and provides better encryption and decryption performance. For more information about the KMS v2 encryption and decryption process, see KMS v2.
To upgrade your cluster from the KMS v1 API to the KMS v2 API, see How do I upgrade to KMS v2 for secret encryption at rest?
The Kubernetes community deprecated the KMS v1 API in version 1.27 and disabled it by default in version 1.29. For compatibility, ACK Pro clusters and still enable the KMS v1 API by using a KMS v1 feature gate.
Enable encryption at rest for an ACK Pro cluster
For new clusters
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click Create Kubernetes Cluster.
-
At the bottom of the ACK Managed Cluster tab, expand Advanced Options (Optional). Find Secret Encryption, select Select Key, and then select a KMS key ID from the drop-down list. Configure other parameters as required and click Create Kubernetes Cluster.
For more information about how to configure an ACK Pro cluster, see Parameters for creating an ACK managed cluster.
Log on to the ActionTrail console and click Cluster Events Query in the left-side navigation pane. If you can find encryption and decryption event logs that use the AliyunCSManagedSecurityRole system role on the Cluster Events Query page, secret encryption at rest is enabled for the cluster.
If you no longer need this feature, go to the Clusters page, click the cluster name, and then click the Cluster Information tab. On the Basic Information page, turn off the Security and Auditing switch in the Secret Encryption section.
For existing clusters
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of the target cluster. On the cluster details page, click the Basic Information tab. In the Security and Auditing section, turn on the Secret Encryption switch.
The first time you enable this feature, you are prompted to grant permissions. Click Go to RAM console. On the Cloud Resource Access Authorization page for Resource Access Management (RAM), click Confirm.
Note-
To enable encryption at rest, make sure the logged-in RAM user or RAM role has RBAC administrator or O&M permissions on the cluster. For more information, see Authorize resources in a cluster by using RBAC.
-
To grant permissions to the AliyunCSManagedSecurityRole role, make sure you are logged in with an Alibaba Cloud account or a RAM user or RAM role that has RAM management permissions.
-
-
In the Secret Encryption dialog box, select an existing KMS key and click OK.
If you do not have a KMS key, click Create Key to go to the Key Management Service console and create a key. For more information, see Create a key.
When the cluster status changes from Updating to Running, secret encryption at rest is enabled for the cluster.
If you no longer need this feature, you can turn off the Security and Auditing switch in the Secret Encryption section.
Automatic key rotation
You can use the automatic key rotation feature of KMS to encrypt secrets at rest. When a key is automatically rotated, existing secrets remain encrypted with the previous key version. New secrets are encrypted with the new key version. For more information about automatic key rotation, see Key rotation.
To ensure that existing secrets are also encrypted with the new key version, run the following command to re-encrypt all existing secrets after the key is automatically rotated.
kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"FAQ
Upgrade to KMS v2
To use KMS v2 to encrypt secrets at rest, make sure that your ACK Pro cluster or is version 1.31 or later.
-
If encryption at rest has not been enabled for your cluster, enable it by following the instructions in Enable encryption at rest for an ACK Pro cluster. After it is enabled, the cluster automatically uses the KMS v2 API to encrypt and decrypt secrets.
-
If encryption at rest is already enabled for your cluster, you can switch to the KMS v2 encryption method during off-peak hours by using either of the following methods:
-
Method 1: Run the
kubectl get secrets --all-namespaces -o json | kubectl replace -f -command. This command re-encrypts all existing secrets by using the KMS v2 API. -
Method 2: Log on to the Container Service for Kubernetes (ACK) console. On the Cluster Information tab of the Basic Information page, find and turn off Secret Encryption. This triggers a re-encryption process for all secrets in the cluster. After a short period, re-enable the feature by following the instructions in Enable encryption at rest for an ACK Pro cluster. After the feature is re-enabled, all secrets in the cluster are encrypted at rest by using the KMS v2 API.
-
Are kubectl-retrieved secrets encrypted?
No. Encryption at rest encrypts the raw secret data in etcd, which is then stored as ciphertext. However, when you use the Secret API through the API server, you receive the plaintext secret data.
How to prevent a RAM user or RAM role from enabling or disabling Secret encryption at rest in an existing ACK Pro cluster
You can attach a RAM permission policy that explicitly denies the cs:UpdateKMSEncryption action to the RAM user or RAM role. This prevents the RAM user or RAM role from enabling or disabling encryption at rest for an existing ACK Pro cluster or . For more information, see Use RAM to grant permissions for clusters and cloud resources.
{
"Action": [
"cs:UpdateKMSEncryption"
],
"Effect": "Deny",
"Resource": [
"*"
]
}