Container Service for Kubernetes:Troubleshoot component errors

Symptoms

The console displays an error message similar to:

Addon status not match, failed upgrade helm addon arms-cmonitor for cluster c3cf94b952cd34b54b71b10b7********, err: rendered manifests contain a resource that already exists. Unable to continue with update: ConfigMap "otel-collector-config" in namespace "arms-prom" exists and cannot be imported into the current release

Cause

A resource required by the component already exists in the cluster, preventing installation. This typically happens when:

• Another version of the component (such as the open-source version) is already installed by a different method.

• The component was installed with Helm V2, and its resources were not removed before migrating to Helm V3.

• A resource with the same name as one required by the component was created manually.

Solution

Delete the conflicting resources identified in the error message, then retry the installation or update.

The error message tells you which resource to delete. The following sections list the exact commands for specific components.

kubectl delete ClusterRole arms-kube-state-metrics
kubectl delete ClusterRole arms-node-exporter
kubectl delete ClusterRole arms-prom-ack-arms-prometheus-role
kubectl delete ClusterRole arms-prometheus-oper3
kubectl delete ClusterRole arms-prometheus-ack-arms-prometheus-role
kubectl delete ClusterRole arms-pilot-prom-k8s
kubectl delete ClusterRoleBinding arms-node-exporter
kubectl delete ClusterRoleBinding arms-prom-ack-arms-prometheus-role-binding
kubectl delete ClusterRoleBinding arms-prometheus-oper-bind2
kubectl delete ClusterRoleBinding kube-state-metrics
kubectl delete ClusterRoleBinding arms-pilot-prom-k8s
kubectl delete ClusterRoleBinding arms-prometheus-ack-arms-prometheus-role-binding
kubectl delete Role arms-pilot-prom-spec-ns-k8s
kubectl delete Role arms-pilot-prom-spec-ns-k8s -n kube-system
kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s
kubectl delete RoleBinding arms-pilot-prom-spec-ns-k8s -n kube-system

kubectl delete MutatingWebhookConfiguration ack-node-local-dns-admission-controller

kubectl delete ConfigMap otel-collector-config -n arms-prom
kubectl delete ClusterRoleBinding arms-prom-cmonitor-role-binding
kubectl delete ClusterRoleBinding arms-prom-cmonitor-install-init-role-binding
kubectl delete ClusterRole arms-prom-cmonitor-role
kubectl delete ClusterRole arms-prom-cmonitor-install-init-role
kubectl delete ServiceAccount cmonitor-sa-install-init -n kube-system

Cause

A Helm release with the same name as the component already exists in the cluster. This prevents Helm from installing or updating the component. Common causes:

• Another version of the component is installed by a different method.

• A leftover Helm release remains from a previous installation attempt.

Solution

1. Log on to the ACK console. In the left navigation pane, click ACK consoleACK consoleACK consoleACK consoleClusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Applications > Helm.

3. Find the Helm release named after the component. In the Actions column, click Delete. In the dialog box, select Clear Release Records and click OK.

4. Install or update the component.

Cause

The update request was submitted, but the component pods cannot reach the Ready state within the timeout period.

Troubleshooting

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Operations > Event Center.

3. On the Events (Cluster Resource Events) tab, set Level to Warning, select the namespace where the component is deployed, and set Type to Pod. Review the event details to identify the cause. The following section describes common causes and their solutions.

Common causes and solutions

Cause

ACK cannot reach the Kubernetes API server. The most common cause is that the Server Load Balancer (SLB) instance exposing the API server is misconfigured or not working as expected.

Solution

See Troubleshoot API server request exceptions.

Cause

Resources required by the component are missing—likely deleted or modified externally—so the component cannot be updated in place.

Solution

Uninstall the component and install the latest version.

Cause

The component uses Helm V2, which depends on Tiller for installation and updates. Tiller has encountered an error and is inaccessible.

Solution

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Workloads > Pods.

3. Select the kube-system namespace. Find and delete the tiller pod. The system automatically recreates it.

4. After the Tiller pod reaches the Ready state, retry the component operation.

Symptoms

The console displays an error message similar to:

failed to create: Internal error occurred: failed calling webhook "rancher.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation?timeout=10s": no endpoints available for service "rancher-webhook"

Cause

A mutating webhook configured for a component resource cannot be called, blocking resource updates.

Solution

Troubleshoot the failing webhook and fix the issue, then update the component again. You can identify the webhook that cannot be called from the error message.

In the example above, the rancher-webhook webhook in the cattle-system namespace is unavailable.

Cause

The cluster uses Helm V2, but Tiller lacks the RBAC permissions needed to query and update resources, preventing component installation or updates.

Solution

Grant the required RBAC permissions to Tiller. For details, see Role-based access control.

Cause

The cluster uses Helm V2, but no Tiller pod is running normally in the cluster.

Solution

Troubleshoot the tiller-deploy pod in the kube-system namespace. After the pod runs normally, retry the component operation. For troubleshooting steps, see Troubleshoot pod issues.

Cause

A ClusterRoleBinding required by the component already exists in the cluster, but its configuration conflicts with what the component expects. This typically occurs when an open-source version of the component is installed separately.

Solution

Uninstall the open-source version of the component from the cluster:

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of the target cluster. In the left pane, choose Applications > Helm.

3. Find the Helm release named after the component. In the Actions column, click Delete. In the dialog box, select Clear Release Records and click OK.

4. Install or update the component.

Symptoms

The console displays an error message similar to:

spec.template.spec.initContainers[1].name: Duplicate value: "install-cni"

Cause

The YAML manifests of the currently installed component version are incompatible with the target version, preventing the update. This can happen when:

• Another version of the component (such as the open-source version) is installed by a different method.

• The component's YAML manifests were modified manually.

• The currently installed version is no longer supported.

Solution

Modify the component's YAML manifests based on the error message. If you need assistance, submit a ticket.

spec.template.spec.initContainers[1].name: Duplicate value: "install-cni"

kubectl -n kube-system edit ds kube-flannel-ds

      containers:
      - name: kube-flannel
        image: registry-vpc.{{.Region}}.aliyuncs.com/acs/flannel:{{.ImageVersion}}
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        ...
        # Irrelevant lines are not shown. Delete comment lines 7 to 21.
    # - command:
      # - /bin/sh
      # - -c
      # - set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf;
        # while true; do sleep 3600; done
      # image: registry-vpc.cn-beijing.aliyuncs.com/acs/flannel:v0.11.0.1-g6e46593e-aliyun
      # imagePullPolicy: IfNotPresent
      # name: install-cni
      # resources: {}
      # terminationMessagePath: /dev/termination-log
      # terminationMessagePolicy: File
      # volumeMounts:
      # - mountPath: /etc/cni/net.d
        # name: cni
      # - mountPath: /etc/kube-flannel/
         # Irrelevant lines are not shown. Delete comment lines 7 to 21.
          name: flannel-cfg
        ...