Restrict worker RAM role permissions

Updated at:
Copy as MD

Restrict the permissions of the Resource Access Management (RAM) role assigned to worker nodes based on the principle of least privilege to improve the security of your ACK managed clusters.

Important

Modifying RAM policies can cause application or add-on failures if required permissions are removed. Test in a staging environment before applying changes to production.

Prerequisites

Before you begin, make sure that you have:

Step 1: Assess current permissions

Determine whether the worker RAM role has excess permissions that need to be restricted.

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of the target cluster. On the Basic Information tab, click the link next to Worker RAM Role.

  3. In the RAM console, go to the Permissions tab of the Role page.

    • If there are no policies attached, no further action is needed. The worker RAM role has no extra permissions.

    • If policies are attached (for example, k8sWorkerRolePolicy-db8ad5c7***), the role may have more permissions than necessary. Evaluate against your business needs based on the principle of least privilege, then proceed to the next step.

Note

Review each attached policy to identify the Alibaba Cloud API actions it grants. This establishes a baseline before you make changes.

Step 2: Upgrade add-ons

Historically, ACK add-ons relied on the node's worker RAM role for cloud resource access. Current ACK add-ons use token-based authentication (managed service roles) instead. Upgrade your cluster add-ons and analyze audit logs to remove unnecessary permissions from the worker RAM role and limit the impact of potential security incidents.

Important

Upgrade add-ons one at a time. Confirm that each upgrade succeeds before starting the next. Read the remarks for each add-on before upgrading.

For more information, see Manage add-ons.

Add-ons installed through Add-ons

On the Add-ons page, upgrade each installed add-on to at least the minimum version shown below. For add-ons that are already at or above the minimum version, redeploy them with the command listed in the table, or redeploy them through the console.

Monitoring and observability

Add-on

Minimum version

Redeploy command

Notes

metrics-server

v0.3.9.4-ff225cd-aliyun

kubectl -n kube-system rollout restart deployment/metrics-server

None

alicloud-monitor-controller

v1.5.5

kubectl -n kube-system rollout restart deployment/alicloud-monitor-controller

None

arms-prometheus

1.1.11

kubectl -n arms-prom rollout restart deployment/arms-prometheus-ack-arms-prometheus

None

ack-cost-exporter

1.0.10

kubectl -n kube-system rollout restart deployment/ack-cost-exporter

Before upgrading, grant permissions for the managed cost role.

ack-node-problem-detector

1.2.16

kubectl -n kube-system rollout restart deployment/ack-node-problem-detector-eventer

None

Networking

Add-on

Minimum version

Redeploy command

Notes

terway

v1.0.10.333-gfd2b7b8-aliyun

kubectl -n kube-system rollout restart daemonset/terway

Upgrade the Terway according to your cluster's Terway mode. For more information, see Use the Terway network plug-in. After upgrading, check the Terway configuration.

terway-eni

v1.0.10.333-gfd2b7b8-aliyun

kubectl -n kube-system rollout restart daemonset/terway-eni

See remarks for terway.

terway-eniip

v1.0.10.333-gfd2b7b8-aliyun

kubectl -n kube-system rollout restart daemonset/terway-eniip

See remarks for terway.

terway-controlplane

v1.2.1

kubectl -n kube-system rollout restart deployment/terway-controlplane

None

mse-ingress-controller

1.1.5

kubectl -n mse-ingress-controller rollout restart deployment/ack-mse-ingress-controller

Before upgrading, grant permissions for the managed Microservices Engine (MSE) role.

Storage

Add-on

Minimum version

Redeploy command

Notes

csi-plugin

v1.18.8.45-1c5d2cd1-aliyun

kubectl -n kube-system rollout restart daemonset/csi-plugin

None

csi-provisioner

v1.18.8.45-1c5d2cd1-aliyun

kubectl -n kube-system rollout restart deployment/csi-provisioner

None

storage-operator

v1.18.8.55-e398ce5-aliyun

kubectl -n kube-system rollout restart deployment/storage-auto-expander
kubectl -n kube-system rollout restart deployment/storage-cnfs
kubectl -n kube-system rollout restart deployment/storage-monitor
kubectl -n kube-system rollout restart deployment/storage-snapshot-manager
kubectl -n kube-system rollout restart deployment/storage-operator

None

alicloud-disk-controller

v1.14.8.51-842f0a81-aliyun

kubectl -n kube-system rollout restart deployment/alicloud-disk-controller

None

flexvolume

v1.14.8.109-649dc5a-aliyun

kubectl -n kube-system rollout restart daemonset/flexvolume

Migrate FlexVolume to CSI.

Logging

Add-on

Minimum version

Redeploy command

Notes

logtail-ds

v1.0.29.1-0550501-aliyun

kubectl -n kube-system rollout restart daemonset/logtail-ds
kubectl -n kube-system rollout restart deployment/alibaba-log-controller

None

loongcollector

v3.0.2

kubectl -n kube-system rollout restart daemonset/loongcollector-ds
kubectl -n kube-system rollout restart deployment/loongcollector-operator

None

Security and image management

Add-on

Minimum version

Redeploy command

Notes

aliyun-acr-credential-helper

v23.02.06.2-74e2172-aliyun

kubectl -n kube-system rollout restart deployment/aliyun-acr-credential-helper

Before upgrading, grant permissions for the managed Container Registry (ACR) role. If you do not have custom RAM permissions and do not need to pull images across accounts, go to Add-ons and set tokenMode to managedRole. If you do not need credential-free image pulling for private images, uninstall this add-on.

ack-onepilot

3.0.11

kubectl -n ack-onepilot rollout restart deployment/ack-onepilot-ack-onepilot

Before upgrading, grant permissions for the managed MSE role.

Cluster Autoscaler (installed through a node pool)

Add-on

Minimum version

Redeploy command

Notes

cluster-autoscaler

v1.3.1-bcf13de9-aliyun

kubectl -n kube-system rollout restart deployment/cluster-autoscaler

To check the current version, use either method:

Check the Terway configuration

If your cluster has terway, terway-eni, or terway-eniip installed, verify that the Terway ConfigMap contains the correct credential path.

  1. Run the following command to edit the Terway ConfigMap:

       kubectl edit cm eni-config -n kube-system
  2. In the eni_conf section, check for the following line:

       "credential_path": "/var/addon/token-config",
    • If this line is present, no change is needed.

    • If this line is missing, add it below the min_pool_size configuration.

  3. Redeploy the Terway workload by running the corresponding redeploy command from the networking table above.

Step 3: Collect audit logs

Set up audit logging to track which Alibaba Cloud API operations the worker RAM role performs. This data tells you which permissions are still in use and which can be safely removed.

Note

Collect audit logs for at least one week.

  1. Log on to the ActionTrail console.

  2. Create a single-account trail for the region where the cluster is located. When creating the trail, select Delivery to Log Service. For more information, see Create a single-account trail.

For the full list of services that ActionTrail supports, see Supported Alibaba Cloud services.

Step 4: Test cluster features

After upgrading add-ons and redeploying workloads, verify that core cluster features still work correctly.

Category

Test case

Reference

Compute

Scale nodes out and in.

Manually scale a node pool

Network

Verify that IP addresses are assigned to pods.

Deployments and releases

Storage

Deploy workloads that use external storage (if applicable).

Storage - CSI

Monitoring

Confirm that monitoring and alert data is available.

Observability

Elasticity

Trigger node autoscaling (if applicable).

Enable node autoscaling

Security

Pull a private image without credentials (if applicable).

Use the aliyun-acr-credential-helper to pull images without using a secret

Important

In addition to the tests above, test the business logic of the applications deployed in your cluster.

Step 5: Analyze audit logs

After collecting logs for at least one week, query the data to identify which API operations the worker RAM role performed.

  1. Log on to the Simple Log Service (SLS) console.

  2. In the Projects section, click the project you specified in Step 3.

    image

  3. On the Log Storage > Logstore tab, click the destination Logstore. The Logstore is named actiontrail_<trail_name>.

  4. Run the following query to list the API operations called through the Security Token Service (STS) token of the worker RAM role. Replace <worker_role_name> with the name of the worker RAM role for your cluster. The results show which Alibaba Cloud services and API operations are being called. Use this data in Step 6 to decide which permissions to keep and which to remove.

       * and event.userIdentity.userName: <worker_role_name> | select "event.serviceName", "event.eventName", count(*) as total GROUP BY "event.eventName", "event.serviceName"

Step 6: Remove unused permissions

Based on the audit log analysis, remove permissions that the worker RAM role does not need.

  1. Log on to the ACK console. In the left navigation pane, click Clusters.

  2. On the Clusters page, click the name of the target cluster. On the Basic Information tab, click the link next to Worker RAM Role.

  3. On the Permissions tab of the Role page, click the destination access policy to go to the Policy Content tab. Click Modify Policy.

    Important

    Before you modify the policy document, back it up. This lets you roll back the permission configuration if needed.

  4. Delete unnecessary permissions based on the audit log analysis from Step 5. For example, delete the Action permissions from the policy that were not invoked during your audit period. If the query returned no results, you can safely detach the entire policy from the role.

  5. Redeploy the workloads using the commands listed in Step 2.

  6. Repeat Steps 4, 5, and 6 until the worker RAM role has only the minimum permissions required by the add-ons or applications.

References