Container Service for Kubernetes:Troubleshoot add-on errors

Symptoms

The console displays an error similar to:

Addon status not match, failed upgrade helm addon arms-cmonitor for cluster c3cf94b952cd34b54b71b10b7********, err: rendered manifests contain a resource that already exists. Unable to continue with update: ConfigMap "otel-collector-config" in namespace "arms-prom" exists and cannot be imported into the current release

Cause

A resource required by the add-on already exists in the cluster. Common causes:

• Another version (such as the open-source version) is installed by a different method.

• The add-on was installed with Helm V2, and its resources were not removed before migrating to Helm V3.

• A resource with the same name was created manually.

Solution

Delete the conflicting resources from the error message, then retry the operation.

The following sections list commands for specific add-ons.

kubectl delete ClusterRole arms-kube-state-metrics
kubectl delete ClusterRole arms-node-exporter
kubectl delete ClusterRole arms-prom-ack-arms-prometheus-role
kubectl delete ClusterRole arms-prometheus-oper3
kubectl delete ClusterRole arms-prometheus-ack-arms-prometheus-role
kubectl delete ClusterRole arms-pilot-prom-k8s
kubectl delete ClusterRoleBinding arms-node-exporter
kubectl delete ClusterRoleBinding arms-prom-ack-arms-prometheus-role-binding
kubectl delete ClusterRoleBinding arms-prometheus-oper-bind2
kubectl delete ClusterRoleBinding kube-state-metrics
kubectl delete ClusterRoleBinding arms-pilot-prom-k8s
kubectl delete ClusterRoleBinding arms-prometheus-ack-arms-prometheus-role-binding
kubectl delete Role arms-pilot-prom-spec-ns-k8s
kubectl delete Role arms-pilot-prom-spec-ns-k8s -n kube-system
kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s
kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s -n kube-system

kubectl delete MutatingWebhookConfiguration ack-node-local-dns-admission-controller

kubectl delete ConfigMap otel-collector-config -n arms-prom
kubectl delete ClusterRoleBinding arms-prom-cmonitor-role-binding
kubectl delete ClusterRoleBinding arms-prom-cmonitor-install-init-role-binding
kubectl delete ClusterRole arms-prom-cmonitor-role
kubectl delete ClusterRole arms-prom-cmonitor-install-init-role
kubectl delete ServiceAccount cmonitor-sa-install-init -n kube-system

Cause

A Helm release with the same name already exists, preventing installation or update. Common causes:

• Another version is installed by a different method.

• A leftover Helm release remains from a previous installation attempt.

Solution

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Applications > Helm.

3. Find the add-on's Helm release. In the Actions column, click Delete. In the dialog box, select Clear Release Records and click OK.

4. Install or update the add-on.

Cause

Add-on pods cannot reach the Ready state within the timeout period after the update is submitted.

Troubleshooting

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Operations > Event Center.

3. On the Events (Cluster Resource Events) tab, set Level to Warning, select the namespace where the add-on is deployed, and set Type to Pod. Review the event details to identify the cause.

Common causes and solutions

Cause

ACK cannot reach the Kubernetes API server, typically because the Server Load Balancer (SLB) instance exposing the API server is misconfigured.

Solution

See Troubleshoot API server request exceptions.

Cause

Required add-on resources are missing—likely deleted or modified externally—preventing in-place update.

Solution

Uninstall the add-on and install the latest version.

Cause

The add-on uses Helm V2, which depends on Tiller. Tiller has encountered an error and is inaccessible.

Solution

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Workloads > Pods.

3. Select the kube-system namespace. Find and delete the tiller pod. The system automatically recreates it.

4. After the Tiller pod reaches the Ready state, retry the add-on operation.

Symptoms

The console displays an error similar to:

failed to create: Internal error occurred: failed calling webhook "rancher.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s": no endpoints available for service "rancher-webhook"

Cause

A mutating webhook for an add-on resource cannot be called, blocking updates.

Solution

Fix the failing webhook identified in the error message, then retry the add-on update.

In the example, the rancher-webhook webhook in the cattle-system namespace is unavailable.

Cause

The cluster uses Helm V2, but Tiller lacks the RBAC permissions to manage resources, preventing component operations.

Solution

Grant the required RBAC permissions to Tiller. See Role-based access control.

Cause

The cluster uses Helm V2, but no Tiller pod is running.

Solution

Troubleshoot the tiller-deploy pod in the kube-system namespace. After the pod runs normally, retry the add-on operation. See Troubleshoot pod issues.

Cause

A ClusterRoleBinding required by the add-on exists but has a conflicting configuration, typically caused by a separately installed open-source version.

Solution

Uninstall the open-source component version:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Applications > Helm.

3. Find the add-on's Helm release. In the Actions column, click Delete. In the dialog box, select Clear Release Records and click OK.

4. Install or update the add-on.

Symptoms

The console displays an error similar to:

spec.template.spec.initContainers[1].name: Duplicate value: "install-cni"

Cause

The YAML manifests of the installed version are incompatible with the target version. Common causes:

• Another version (such as the open-source version) is installed by a different method.

• The add-on's YAML manifests were modified manually.

• The currently installed version is no longer supported.

Solution

Modify the component's YAML manifests based on the error message. For assistance, submit a ticket.

spec.template.spec.initContainers[1].name: Duplicate value: "install-cni"

kubectl -n kube-system edit ds kube-flannel-ds

      containers:
      - name: kube-flannel
        image: registry-vpc.{{.Region}}.aliyuncs.com/acs/flannel:{{.ImageVersion}}
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        ...
        # Irrelevant lines are not shown. Delete comment lines 7 to 21.
    # - command:
      # - /bin/sh
      # - -c
      # - set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf;
        # while true; do sleep 3600; done
      # image: registry-vpc.cn-beijing.aliyuncs.com/acs/flannel:v0.11.0.1-g6e46593e-aliyun
      # imagePullPolicy: IfNotPresent
      # name: install-cni
      # resources: {}
      # terminationMessagePath: /dev/termination-log
      # terminationMessagePolicy: File
      # volumeMounts:
      # - mountPath: /etc/cni/net.d
        # name: cni
      # - mountPath: /etc/kube-flannel/
         # Irrelevant lines are not shown. Delete comment lines 7 to 21.
          name: flannel-cfg
        ...