ack-secret-manager syncs secrets from Key Management Service (KMS) to your Container Service for Kubernetes (ACK) cluster as Kubernetes Secrets. Applications mount these Secrets through the file system to access up-to-date KMS credentials without managing them directly.
How it works
Two custom resource definitions (CRDs) drive the sync process:
SecretStore — defines how ack-secret-manager authenticates to KMS
ExternalSecret — defines which KMS secrets to sync and where to store them in the cluster
When ack-secret-manager processes an ExternalSecret, it reads the referenced SecretStore for authentication credentials, retrieves the specified KMS secret, and creates or updates a Kubernetes Secret with the same name and namespace as the ExternalSecret.
Security considerations
Before enabling secret synchronization, assess these risks:
File system traversal: Secrets mounted via the file system can be exposed if an attacker exploits CVE vulnerabilities to traverse cluster directories.
Debug and log exposure: Misconfigured debug breakpoints or overly permissive log access can leak Secrets. Avoid referencing Secrets through environment variables.
Excessive permissions: Apply the least privilege principle when granting ack-secret-manager access to KMS.
If your application doesn't need to persist secrets locally, grant pods minimal permissions via RRSA and call GetSecretValue directly from the application. This avoids exposing secrets in the pod file system or Kubernetes Secrets.
Prerequisites
Before you begin, ensure that you have:
An ACK cluster created in the same region as the KMS secret you want to sync. Supported cluster types:
Step 1: Install ack-secret-manager
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the target cluster name. In the left navigation pane, choose Applications > Helm.
On the Helm page, click Deploy. In the Chart section of the Deploy panel, find and select ack-secret-manager, then click Next. In the Confirm dialog, the chart installs to the
kube-systemnamespace by default. Click Yes to proceed. To use a custom namespace, configure Application Name and Namespace in the Basic Information step.In the Parameters step, select the latest chart version and configure the following parameters as needed, then click OK. RRSA authentication (recommended): Set
rrsa.enabletotrueto enable RAM Roles for Service Accounts (RRSA). This allows ack-secret-manager to authenticate using a service account identity without managing AccessKey pairs. Scheduled sync: Configure the following parameters to enable periodic secret synchronization. Throttling (for clusters with many ExternalSecrets): Setcommand.maxConcurrentKmsSecretPullsto limit concurrent KMS secret pulls per second. The default value is10. Without this limit, a large number of ExternalSecrets can trigger throttling on the KMS or RAM side. KMS endpoint: Setcommand.kmsEndpointto specify a global KMS endpoint for all secret requests. This is a global configuration; you can also configure endpoints at the individual secret level. For details, see Configure KMS endpoint addresses.Parameter Description command.disablePollingWhether to disable periodic polling. Set to falseto enable periodic sync.command.pollingIntervalSync frequency. For example, 120ssyncs secrets every 2 minutes.



After installation, the ACK console redirects to the ack-secret-manager page. Verify the installation by checking that the expected resources are created.

Step 2: Authorize ack-secret-manager to access KMS
Create a SecretStore to define how ack-secret-manager authenticates to KMS. Without a SecretStore, ack-secret-manager cannot sync secrets from KMS.
Choose the authorization method based on your cluster type:
| Method | Applicable cluster types | Key advantage |
|---|---|---|
| RRSA (recommended) | ACK managed clusters, ACK Serverless clusters (Kubernetes 1.22+) | No AccessKey pairs to manage |
| Worker RAM role | ACK managed clusters, ACK dedicated clusters, registered clusters | Simple setup; no additional credentials |
| AccessKey pair | All cluster types | Works universally |
Use RRSA to grant permissions
RRSA enables pod-level permission control without AccessKey pairs. It applies to ACK managed clusters and ACK Serverless clusters running Kubernetes 1.22 or later.
Enable RRSA for the ACK cluster in the ACK console. This creates an OpenID Connect (OIDC) identity provider (IdP) for the cluster. For details, see Enable RRSA.
Create a RAM role with the IdP as the trusted entity. Set Principal Type to Identity Provider and configure the following parameters. For details, see Create a RAM role for an OIDC IdP.
Parameter Value Identity provider type Select OIDC Identity provider Select ack-rrsa-<cluster_id>, where<cluster_id>is your cluster IDCondition — oidc:iss Keep the default Condition — oidc:aud Keep the default Condition — oidc:sub Add manually: Key = oidc:sub, Operator =StringEquals, Value =system:serviceaccount:<namespace>:<serviceAccountName>. For example:system:serviceaccount:kube-system:ack-secret-manager. If you installed ack-secret-manager in a namespace other thankube-system, use the actual namespace.Create a custom RAM policy that grants access to KMS secrets, then attach it to the RAM role.
Create a custom policy with the following content. For details, see Create custom policies. ``
json { "Action": [ "kms:GetSecretValue", "kms:Decrypt" ], "Resource": [ "*" ], "Effect": "Allow" }``Attach the policy to the RAM role. For details, see Grant permissions to a RAM role.
Create a SecretStore using RRSA.
Create a file named
secretstore-rrsa.yamlwith the following content. Replace the placeholders as described.Placeholder Description {accountID}ID of the Alibaba Cloud account used to sync KMS secrets {clusterID}ID of your ACK cluster {roleName}Name of the RAM role created in step 2 apiVersion: 'alibabacloud.com/v1alpha1' kind: SecretStore metadata: name: scdemo-rrsa spec: KMS: KMSAuth: oidcProviderARN: "acs:ram::{accountID}:oidc-provider/ack-rrsa-{clusterID}" ramRoleARN: "acs:ram::{accountID}:role/{roleName}"Apply the SecretStore: ``
bash kubectl apply -f secretstore-rrsa.yaml``
Grant permissions to the worker RAM role
This method applies to ACK managed clusters, ACK dedicated clusters, and registered clusters. It does not apply to ACK Serverless clusters, which have no worker RAM role.
Create a custom RAM policy with the following content. For details, see Create custom policies.
{ "Action": [ "kms:GetSecretValue", "kms:Decrypt" ], "Resource": [ "*" ], "Effect": "Allow" }Attach the policy to the worker RAM role of the cluster. For details, see Grant permissions to the worker RAM role.
When using the worker RAM role, you can omit the
secretStoreReffield in the ExternalSecret.
Specify an AccessKey pair to assume a RAM role
This method applies to all ACK cluster types.
Create a RAM role with your Alibaba Cloud account as the trusted entity. For details, see Create a RAM role for a trusted Alibaba Cloud account.
When selecting the trusted Alibaba Cloud account, select Current Account.
Create a custom RAM policy that grants access to KMS secrets, then attach it to the RAM role.
Create a custom policy with the following content. For details, see Create custom policies. ``
json { "Action": [ "kms:GetSecretValue", "kms:Decrypt" ], "Resource": [ "*" ], "Effect": "Allow" }``Attach the policy to the RAM role. For details, see Grant permissions to a RAM role.
Create a custom RAM policy that allows a RAM user to assume the RAM role, then attach it to the RAM user.
Create a policy with the following content. Replace
<account-id>with the Alibaba Cloud account ID that owns the RAM role, and replace<role-name>with the RAM role name. For details on finding the role ARN, see How do I view the ARN of a RAM role?{ "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": "acs:ram:*:<account-id>:role/<role-name>" } ], "Version": "1" }Attach the policy to the RAM user. For details, see Grant permissions to RAM users.
Create a Kubernetes Secret to store the RAM user's AccessKey pair.
Create a file named
ramuser.yamlwith the following content. Replace the AccessKey ID and AccessKey Secret with their Base64-encoded values.apiVersion: v1 data: accessKey: {AccessKey ID encoded in Base64} accessKeySecret: {AccessKey secret encoded in Base64} kind: Secret metadata: name: ramuser namespace: kube-system type: OpaqueApply the Secret: ``
bash kubectl apply -f ramuser.yaml``
Create a SecretStore using the AccessKey pair.
Create a file named
secretstore-ramrole.yamlwith the following content. Replace the placeholders as described.Placeholder Description {accountID}ID of the Alibaba Cloud account used to sync KMS secrets {roleName}Name of the RAM role created in step 1 {secretName}Name of the Kubernetes Secret storing the AccessKey pair {secretNamespace}Namespace of the Kubernetes Secret {secretKey}Key in the Kubernetes Secret that stores the AccessKey value {roleSessionName}Role session name — any custom string apiVersion: 'alibabacloud.com/v1alpha1' kind: SecretStore metadata: name: scdemo-ramrole spec: KMS: KMSAuth: accessKey: name: {secretName} namespace: {secretNamespace} key: {secretKey} accessKeySecret: name: {secretName} namespace: {secretNamespace} key: {secretKey} ramRoleARN: "acs:ram::{accountID}:role/{roleName}" ramRoleSessionName: {roleSessionName}Apply the SecretStore: ``
bash kubectl apply -f secretstore-ramrole.yaml``
Step 3: Sync KMS secrets to the cluster
Create an ExternalSecret to specify which KMS secrets to sync. ack-secret-manager creates a Kubernetes Secret with the same name and namespace as the ExternalSecret.
The namespace and name of the resulting Kubernetes Secret match the ExternalSecret exactly.
Create a file named
external.yamlwith the following content. Replace the placeholders as described.Placeholder Description {KMS secret name}Required. Name of the KMS secret to sync. {Kubernetes secret key}Required. Key name in the Kubernetes Secret where the KMS secret value is stored. {KMS secret version stage}Optional. Version stage label of the KMS secret, such as ACSCurrent. To use a version ID instead, replaceversionStagewithversionId. RDS, PolarDB, Redis/Tair, RAM, and ECS secrets only supportACSPreviousandACSCurrentand ignoreversionId. For details, see Secret elements.{KMS Service endpoint address}Optional. KMS request endpoint. Overrides global and default endpoint configurations. For supported endpoint formats, see Configure KMS endpoint addresses. {secret store name}Optional. Name of the SecretStore created in step 2. Not required when using the worker RAM role. {secret store namespace}Optional. Namespace of the SecretStore. Not required when using the worker RAM role. apiVersion: 'alibabacloud.com/v1alpha1' kind: ExternalSecret metadata: name: esdemo spec: provider: kms # Sync from KMS. Can be omitted or set to kms. data: - key: {KMS secret name} name: {Kubernetes secret key} versionStage: {KMS secret version stage} kmsEndpoint: {KMS Service endpoint address} secretStoreRef: # Omit when using the worker RAM role. name: {secret store name} namespace: {secret store namespace}Apply the ExternalSecret:
kubectl apply -f external.yamlRun the following command to check whether a Kubernetes Secret exists:
kubectl get secret esdemoIf a Kubernetes Secret exists, the KMS secret is synchronized from KMS to your cluster.
Advanced configurations
Sync KMS secrets across Alibaba Cloud accounts
Use this configuration when the KMS instance belongs to one Alibaba Cloud account (Account A) but the ACK cluster belongs to another (Account B). ack-secret-manager uses RRSA to assume a RAM role in Account A via an OIDC provider in Account B, then imports the KMS secret into the cluster.
Configure permissions for Account A
Use Account A to create a RAM role with Account B as the trusted entity. Select Other Account and enter the ID of Account B. For details, see Create a RAM role for a trusted Alibaba Cloud account.
Create a custom RAM policy that grants access to KMS secrets:
{ "Action": [ "kms:GetSecretValue", "kms:Decrypt" ], "Resource": [ "*" ], "Effect": "Allow" }Attach the policy to the RAM role. For details, see Grant permissions to a RAM role.
Configure permissions for Account B
Enable RRSA for the ACK cluster in the ACK console. When installing ack-secret-manager, set
rrsa.enabletotrue. For details, see Enable RRSA.Create a RAM role in Account B with the OIDC IdP as the trusted entity. Set Principal Type to Identity Provider and configure the following parameters. For details, see Create a RAM role for an OIDC IdP.
Parameter Value Identity provider type Select OIDC Identity provider Select ack-rrsa-<cluster_id>, where<cluster_id>is your cluster IDCondition — oidc:iss Keep the default Condition — oidc:aud Keep the default Condition — oidc:sub Add manually: Key = oidc:sub, Operator =StringEquals, Value =system:serviceaccount:kube-system:ack-secret-managerCreate a custom RAM policy in Account B that allows assuming the RAM role in Account A, then attach it to the Account B RAM role.
Create a policy with the following content. Replace
<account-id>with the ID of Account A and<role-name>with the RAM role created in Account A. For details on finding the ARN, see How do I view the ARN of a RAM role?{ "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Resource": "acs:ram:*:<account-id>:role/<role-name>" } ], "Version": "1" }Attach the policy to the RAM role created in Account B.
Create a SecretStore for cross-account access.
Create a file named
secretstore-ramrole.yamlwith the following content. Replace the placeholders as described.Placeholder Description {ACK-accountID}ID of Account B (the account with the ACK cluster) {clusterID}ID of your ACK cluster {ACK-roleName}Name of the RAM role created in Account B {KMS-accountID}ID of Account A (the account with the KMS instance) {KMS-roleName}Name of the RAM role created in Account A {roleSessionName}Role session name — any custom string apiVersion: 'alibabacloud.com/v1alpha1' kind: SecretStore metadata: name: scdemo-cross-account spec: KMS: KMSAuth: oidcProviderARN: "acs:ram::{ACK-accountID}:oidc-provider/ack-rrsa-{clusterID}" ramRoleARN: "acs:ram::{ACK-accountID}:role/{ACK-roleName}" remoteRamRoleARN: "acs:ram::{KMS-accountID}:role/{KMS-roleName}" remoteRamRoleSessionName: {roleSessionName}
Create an ExternalSecret to specify the sync details. For details, see Step 3: Sync KMS secrets to the cluster.
Parse JSON and YAML secrets
Extract specific fields from a JSON secret
Use jmesPath to extract specific key-value pairs from a KMS secret stored in JSON format.
For example, given the following JSON secret stored in KMS Secrets Manager:
{"name":"tom","friends":[{"name":"lily"},{"name":"mark"}]}Create an ExternalSecret with jmesPath to extract specific fields:
path: JMESPath expression to extract the field. Required. See the JMESPath specification.objectAlias: Key name in the resulting Kubernetes Secret.
apiVersion: 'alibabacloud.com/v1alpha1'
kind: ExternalSecret
metadata:
name: es-json-demo
spec:
provider: kms
data:
- key: {KMS secret name}
versionStage: {KMS secret version stage}
secretStoreRef:
name: {secret store name}
namespace: {secret store namespace}
jmesPath:
- path: "name" # Extracts: "tom"
objectAlias: "myname"
- path: "friends[0].name" # Extracts: "lily"
objectAlias: "friendname"Auto-parse a JSON secret
If the JSON structure is unknown or variable, use dataProcess.extract to automatically parse all key-value pairs from a JSON secret. Use dataProcess.replaceRule to rename keys that don't conform to Kubernetes Secret key naming requirements.
For example, given the following JSON secret with non-conforming key names:
{"/name-invalid":"lily","name-invalid/":[{"name":"mark"}]}apiVersion: 'alibabacloud.com/v1alpha1'
kind: ExternalSecret
metadata:
name: extract-secret
spec:
provider: kms
dataProcess:
- extract:
key: {KMS secret name}
versionStage: ACSCurrent
secretStoreRef:
name: {secret store name}
namespace: {secret store namespace}
replaceRule:
- source: "^/.*d$" # Keys starting with / and ending with d -> "tom"
target: "tom"
- source: "^n.*/$" # Keys starting with n and ending with / -> "mark"
target: "mark"Extract specific fields from a YAML secret
Use jmesPath to extract specific key-value pairs from a KMS secret stored in YAML format.
For example, given the following YAML secret stored in KMS Secrets Manager:
name: tom
friends:
- name: lily
- name: markapiVersion: 'alibabacloud.com/v1alpha1'
kind: ExternalSecret
metadata:
name: es-yaml-demo
spec:
provider: kms
data:
- key: {KMS secret name}
versionStage: {KMS secret version stage}
secretStoreRef:
name: {secret store name}
namespace: {secret store namespace}
jmesPath:
- path: "name" # Extracts: "tom"
objectAlias: "myname"
- path: "friends[0].name" # Extracts: "lily"
objectAlias: "friendname"Auto-parse a YAML secret
If the YAML structure is unknown or variable, use dataProcess.extract to auto-parse all key-value pairs. Use dataProcess.replaceRule to handle non-conforming key names.
For example, given the following YAML secret:
/name-invalid: lily
name-invalid/:
- name: markapiVersion: 'alibabacloud.com/v1alpha1'
kind: ExternalSecret
metadata:
name: extract-secret
spec:
provider: kms
dataProcess:
- extract:
key: {KMS secret name}
versionStage: ACSCurrent
secretStoreRef:
name: {secret store name}
namespace: {secret store namespace}
replaceRule:
- source: "^/.*d$" # Keys starting with / and ending with d -> "tom"
target: "tom"
- source: "^n.*/$" # Keys starting with n and ending with / -> "mark"
target: "mark"Configure KMS endpoint addresses
ack-secret-manager resolves the KMS endpoint using the following priority order:
| Type | Field | Priority | Description |
|---|---|---|---|
| Secret-level | ExternalSecret.spec.data.kmsEndpoint | Highest | Applies to a specific secret; overrides global and default configurations. |
| Global | command.kmsEndpoint (startup parameter) | Medium | Applies to all KMS requests that don't have a secret-level endpoint. |
| Default | None | Lowest | Uses kms-vpc.{region}.aliyuncs.com, where {region} is the region of the KMS secret. |
The following example shows how secret-level and global endpoints interact:
apiVersion: "alibabacloud.com/v1alpha1"
kind: ExternalSecret
metadata:
name: esdemo
spec:
provider: kms
data:
- key: test-hangzhou
name: hangzhou-vpc
versionId: v1
# No kmsEndpoint: uses global config if set, otherwise uses kms-vpc.{region}.aliyuncs.com
- key: test-hangzhou
name: hangzhou-public
versionId: v1
kmsEndpoint: kms.cn-hangzhou.aliyuncs.com # Overrides global and defaultSupported endpoint formats:
| Gateway type | Domain type | Endpoint format | Requirements |
|---|---|---|---|
| Dedicated gateway | KMS private domain | {kms-instance-id}.cryptoservice.kms.aliyuncs.com | KMS secret instance and cluster in the same region and VPC. KMS instance version 3.0 or later. |
| Shared gateway | VPC domain | kms-vpc.{region}.aliyuncs.com | KMS secret and cluster in the same region. This is the default endpoint. |
| Shared gateway | Public domain | kms.{region}.aliyuncs.com | Cluster can access the public internet. |
For more information, see Differences between shared and dedicated gateways for accessing KMS.
What's next
To protect Kubernetes Secrets synced from KMS and stored in etcd, enable Secret encryption. For details, see Use KMS to encrypt Kubernetes Secrets.