Container Service for Kubernetes:Manage the csi-plugin and csi-provisioner components

Startup failure: exec /usr/bin/plugin.csi.alibabacloud.com: exec format error

Symptom

The csi-plugin container in the csi-plugin Pod fails to start, and the log shows the error: exec /usr/bin/plugin.csi.alibabacloud.com: exec format error.

Cause

This error usually indicates a CPU architecture mismatch. However, csi-plugin supports both amd64 and arm64 architectures by default. Therefore, if this error occurs on a supported node, the most likely cause is a corrupted CSI image file.

This can happen if the image pull process is interrupted, for example, by a forced shutdown. This leaves an incomplete image file on the node. Although the image metadata exists, the binary executable is invalid or damaged, which prevents the csi-plugin container from starting.

Solution

• Scale out by adding a new node, and then drain the problematic node.

• If you must keep the current node, perform the following steps:

  1. Drain the node to evict its workloads, and then remove the node from the cluster.

  2. Log on to the node and delete all containers, if any.

  3. Delete all files from the /var/lib/containerd directory.

  4. Add the node back to the cluster.

Storage component OOM issues

The sidecar container in the csi-provisioner component pod caches resource information such as pods, PVs, and PVCs. In a large-scale cluster, this can consume a significant amount of memory and might cause the container to experience an Out-of-Memory (OOM) error.

• Managed csi-provisioner: Submit a ticket for assistance.

• Unmanaged csi-provisioner: If an OOM error occurs, adjust the resource limits based on your cluster size.

  1. On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons.

  2. Locate the csi-provisioner component, click the icon, and then click View in YAML.

  3. Edit the component YAML to adjust the resource limits according to your cluster size.

     containers:
         - name: external-disk-provisioner
           image: registry-vpc.cn-huhexxxx.yuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
           resources:
             requests:
               cpu: 10m
               memory: 16Mi
             limits:
               cpu: 500m
               memory: 1024Mi

csi-plugin high network traffic

Symptom

In the cluster pod monitoring dashboard, the csi-plugin component pods show high network traffic.

Cause

This is expected behavior. The csi-plugin component manages the mounting of NAS storage volumes on nodes. When an application pod reads or writes data through a NAS mount point, the resulting network traffic passes through the network namespace of the csi-plugin pod. Therefore, the monitoring system attributes this traffic to the csi-plugin pod.

Solution

No action is required; you can safely ignore this. The traffic is only attributed to the csi-plugin pod for accounting purposes and does not actually consume its resources, nor does it result in double traffic or duplicate billing. This is a monitoring artifact that does not affect performance.

The csi-provisioner component logs report the failed to renew lease xxx timed out waiting for the condition error

Symptom

Run the kubectl logs csi-provisioner-xxxx -n kube-system command to view the CSI logs. The failed to renew lease xxx timed out waiting for the condition error is reported.

Cause

The csi-provisioner component uses a multi-replica model for high availability. To ensure that only one instance serves requests at a time, the replicas use the Kubernetes Lease mechanism for leader election. This mechanism depends heavily on stable communication with the API server.

This error indicates that csi-provisioner cannot access the API server, causing the leader election process to fail. As a result, no replica can become the leader and provide service.

Solution

Check the cluster's network connectivity and the status of the API server. If the issue persists, submit a ticket for assistance.

csi-plugin pre-upgrade check failure

kubectl -nkube-system get ds csi-plugin flexvolume

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
csi-plugin   4         4         4       4            4           kubernetes.io/os=linux   7d
flexvolume   4         4         4       4            4           kubernetes.io/os=linux   7d                                  

csi-plugin upgrade fails after pre-check

The csi-plugin component is deployed as a DaemonSet, which requires all nodes to be in a healthy state. The upgrade fails if the cluster has any nodes in a NotReady or non-Running state.

You must first repair the faulty node and then retry the upgrade. If you cannot identify the cause, contact us to request an assisted manual upgrade.

csi-provisioner missing from console

Earlier versions of csi-provisioner (1.14 and earlier) are deployed as a StatefulSet. If StatefulSet instances of csi-provisioner still exist in the current cluster, you must manually delete them (kubectl delete sts csi-provisioner) before reinstalling.

If you encounter issues during this operation, contact us to request an assisted manual upgrade.

csi-provisioner pre-upgrade check failure

kubectl -nkube-system get ds csi-plugin flexvolume

NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
csi-plugin   4         4         4       4            4           kubernetes.io/os=linux   7d
flexvolume   4         4         4       4            4           kubernetes.io/os=linux   7d                                  

csi-provisioner upgrade fails after pre-check

Contact us to request an assisted manual upgrade.

csi-provisioner upgrade failure: Node count or permissions

Symptom

• Scenario 1: The pre-upgrade check for the csi-provisioner component fails, indicating that the cluster does not have enough nodes.

• Scenario 2: The pre-check for the csi-provisioner component and the upgrade are successful. However, the component's Pod enters the CrashLoopBackOff state, and the logs show a 403 Forbidden error.

  time="2023-08-05T13:54:00+08:00" level=info msg="Use node id : <?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n         \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n <head>\n  <title>403 - Forbidden</title>\n </head>\n <body>\n  <h1>403 - Forbidden</h1>\n </body>\n</html>\n"

Cause and solution

• Scenario 1

  • Cause: The csi-provisioner component uses an active-standby deployment model, and its two replicas must be scheduled to different nodes to satisfy anti-affinity requirements. A single-node cluster cannot meet this scheduling condition, causing the second replica to remain in a Pending state and preventing the upgrade from completing successfully.

  • Solution: Add at least one more node to the cluster to ensure that the multiple replicas can be scheduled correctly.

• Scenario 2

  • Cause: The csi-provisioner component must access the node's instance metadata to obtain information such as the instance ID and region, which are required for calling Alibaba Cloud APIs. If security hardening is enabled on the node or strict network policies are configured, access to the metadata service might be blocked.

  • Solution: Adjust or disable the security hardening policies on the node to ensure the pod can access the metadata service. Also, check security group rules, OS firewall settings, and any relevant security scripts.

csi-provisioner upgrade failure: StorageClass modification

Symptom

The installation or upgrade of csi-provisioner fails its pre-check, indicating that the StorageClass properties do not meet expectations.

Cause

The default StorageClass provided by ACK has a fixed configuration, and its core fields, such as provisioner and parameters, cannot be changed. Manually deleting and recreating a StorageClass with the same name will cause a validation failure and interrupt the component upgrade process.

The CSI installer strictly validates the integrity of these resources to ensure consistent component behavior.

Solution

1. Delete the default StorageClass objects from the cluster, including alicloud-disk-essd, alicloud-disk-available, alicloud-disk-efficiency, alicloud-disk-ssd, and alicloud-disk-topology.

   This operation does not affect the normal use of existing PVs and PVCs.

2. After you delete the StorageClass objects, reinstall csi-provisioner. The installer automatically recreates the StorageClass objects. No further action is needed.

If you need a custom StorageClass, always create a new one with a different name. Do not modify or recreate the default StorageClass objects.