Use KMS to encrypt Kubernetes Secrets in ACS clusters - Container Compute Service

Kubernetes Secrets store sensitive data such as passwords, TLS certificates, and Docker image credentials. By default, Kubernetes stores Secret data in the cluster's etcd. Alibaba Cloud Container Compute Service (ACS) integrates with Key Management Service (KMS) to encrypt all Secret data at rest in etcd, providing an additional layer of protection for sensitive information.

How it works

ACS clusters use the community KMS v2 interface. With KMS v2, the API server encrypts Secret data with a data encryption key (DEK) before writing it to etcd. The DEK itself is encrypted by a KMS key (key encryption key). This envelope encryption model delivers strong encryption and decryption performance at scale.

ACS clusters support default keys, software-protected keys, and hardware-protected keys. For details on managing KMS keys, see Getting started with Key Management. For pricing, see Billing.

For more information, see Using a KMS provider for data encryption and Use KMS keys for envelope encryption.

Prerequisites

Before you begin, make sure that you have:

An ACS cluster of version 1.28 or later (or plan to create one)
A KMS key in the same region as the ACS cluster, created in the KMS console
The required permissions for your account type: Alibaba Cloud account Grant the AliyunCCManagedSecurityRole system role to the Container Compute Service account. If the role has not been granted, a prompt appears when Secret encryption is enabled. You can also go to the Cloud Resource Access Authorization page to grant permissions. RAM user or RAM role
- Administrator or operational permissions for the cluster based on role-based access control (RBAC). See Grant RBAC permissions to RAM users or RAM roles.
- The AliyunKMSCryptoAdminAccess system permission. See Attach a RAM policy to a RAM user or RAM role.

Important

Encryption at rest for Secrets is supported only in ACS clusters of version 1.28 or later.

Enable Secret encryption

Important

After you enable encryption at rest, do not disable or delete the KMS key used for Secret encryption in the KMS console or through an API operation. Doing so makes the cluster API server unavailable, and objects such as Secrets and ServiceAccounts can no longer be retrieved. This disrupts running applications.

Enable during cluster creation

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click Create Kubernetes Cluster in the upper-left corner.
At the bottom of the page, click Show Advanced Options. Select the Select Key checkbox next to Secret Encryption, then choose a KMS key ID from the dropdown list. Configure other parameters as required, then click Confirm Order. For details, see Create an ACS cluster.

Enable for an existing cluster

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster. In the left navigation pane, choose Cluster Information.
On the Cluster Information page, click the Basic Information tab. In the Security and Auditing section, click Enable next to Secret Encryption.
In the Secret Encryption dialog box, select an existing KMS key and click OK. If no KMS key is available, click create keys to go to the KMS console and create one. For more information, see Create a key.

When the cluster status changes from Updating to Running, Secret encryption is enabled.

Verify encryption status

After you enable Secret encryption, verify that it is active:

Log on to the ActionTrail console and click Event Query in the left navigation pane.
Check for encryption and decryption event logs from the aliyuncsmanagedsecurityrole system role. The presence of these events confirms that encryption at rest for Secrets is active.

Rotate the KMS key

KMS supports automatic key rotation. When a key is automatically rotated:

Existing Secrets remain encrypted with the previous key version.
New Secrets are encrypted with the new key version.

For more information, see Configure key rotation.

After the key rotates, re-encrypt all existing Secrets with the new key version by running the following command:

kubectl get secrets --all-namespaces -o json | kubectl annotate --overwrite -f - encryption-key-rotation-time="$(date -u +'%Y-%m-%dT%H:%M:%S%z')"

Disable Secret encryption

Disable for a cluster created with encryption

Log on to the ACS console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the cluster.
On the Cluster Information page, click the Basic Information tab. In the Security and Auditing section, click Close next to Secret Encryption.

Disable for an existing cluster

In the Security and Auditing section of the Basic Information tab, click Close next to Secret Encryption.

FAQ

Does kubectl return ciphertext from etcd?

No. Encryption at rest encrypts the raw Secret data within etcd, storing it as ciphertext. When you retrieve a Secret through the API server, the data is decrypted and returned as plaintext. Commands such as kubectl get secret always return plaintext values.

How do I prevent RAM users from changing encryption settings?

To prevent a RAM user or RAM role from enabling or disabling encryption at rest for Secrets in an existing ACS cluster, attach the following RAM policy with a Deny effect. See Attach a RAM policy to a RAM user or RAM role.

{
    "Action": [
        "cs:UpdateKMSEncryption"
    ],
    "Effect": "Deny",
    "Resource": [
        "*"
    ]
}