FAQ about FlexVolume - Container Service for Kubernetes - Alibaba Cloud Documentation Center


Type	Question
FAQ about storage	What do I do when a volume fails to be mounted? How do I view the log related to storage services? What do I do if I find kubelet contains a pod log that is not managed by ACK?
FAQ about disk volumes	Why does a timeout error occur when I mount a disk to a node? Why does a zone error occur when I mount a disk to an Elastic Compute Service (ECS) instance? Why does an input/output error occur in a disk after a system upgrade? Why does the system prompt The specified disk is not a portable disk when I unmount a disk? Why does the system prompt had volume node affinity conflict when I launch a pod that has a disk mounted? Why does the system prompt can't find disk when I launch a pod that has a disk mounted? Why does the system prompt disk size is not supported when I dynamically provision a PV?
FAQ about NAS volumes	Why does it require a long time to mount a NAS volume? Why does a timeout error occur when I mount a NAS volume? Why does the system prompt chown: option not permitted when I mount a NAS volume? What do I do if I fail to mount a NAS volume? What do I do if the task queue of alicloud-nas-controller is full and PVs cannot be created when I use a dynamically provisioned NAS volume?
FAQ about OSS volumes	Why do I fail to mount an OSS volume? Why does the directory in an OSS volume that is mounted to a container become unavailable after the cluster that runs the pod is upgraded? Why does it require a long time to mount an OSS volume?

What do I do when a volume fails to be mounted?

Check whether FlexVolume and the plug-in for dynamic volume provisioning are installed.

Method 1: Check whether FlexVolume is installed

Run the following command to query information about pods:

kubectl get pod -n kube-system | grep flexvolume

Expected output:

flexvolume-4wh8s            1/1       Running   0          8d
flexvolume-65z49            1/1       Running   0          8d
flexvolume-bpc6s            1/1       Running   0          8d
flexvolume-l8pml            1/1       Running   0          8d
flexvolume-mzkpv            1/1       Running   0          8d
flexvolume-wbfhv            1/1       Running   0          8d
flexvolume-xf5cs            1/1       Running   0          8d

Check whether the FlexVolume pods are in the Running state. Check whether the number of running FlexVolume pods equals the number of nodes in the cluster.

If the FlexVolume pods are not in the Running state, troubleshoot the issue based on the log of the FlexVolume plug-in.

Method 2: Check whether the plug-in for dynamic volume provisioning is installed

To mount a dynamically provisioned disk volume, you must install the plug-in for dynamic volume provisioning. Run the following command to query the pod on which the plug-in runs:

kubectl get pod -n kube-system | grep alicloud-disk

Expected output:

alicloud-disk-controller-8679c9fc76-lq6zb     1/1 Running   0   7d

If the pod is not in the Running state, troubleshoot the issue based on the log of the plug-in.

How do I view the log related to storage services?

You can view the logs of FlexVolume, the provisioner plug-in, and kubelet.

Method 1: Print the log of FlexVolume on Master Node 1

Run the following command to query the pod on which an error occurs:

kubectl get pod -n kube-system | grep flexvolume

Run the following commands in sequence to print the log of the pod on which the error occurs:

kubectl logs flexvolume-4wh8s -n kube-system
kubectl describe pod flexvolume-4wh8s -n kube-system

Note The last several entries of the returned pod information indicate the status of the pod. You can analyze the error based on the log data.

View the logs of the disk driver, NAS driver, and OSS driver.

Perform the following steps to view the logs on the host node. If no PV is mounted to a pod, query the IP address of the node that hosts the pod.

kubectl describe pod nginx-97dc96f7b-xbx8t | grep Node

Expected output:

Node: cn-hangzhou.i-bp19myla3uvnt6zi****/192.168.XX.XX
Node-Selectors:  <none>

Log on to the node and print the logs of the disk driver, NAS driver, and OSS driver.

ssh 192.168.XX.XX
ls /var/log/alicloud/flexvolume*

Expected output:

flexvolume_disk.log  flexvolume_nas.log  flexvolume_o#ss.log

Method 2: Print the log of the provisioner plug-in on Master Node 1

Run the following command to query the pod on which an error occurs:

kubectl get pod -n kube-system | grep alicloud-disk

Run the following commands in sequence to print the log of the pod on which the error occurs:

kubectl logs alicloud-disk-controller-8679c9fc76-lq6zb -n kube-system
kubectl describe pod alicloud-disk-controller-8679c9fc76-lq6zb -n kube-system

Note The last several entries of the returned pod information indicate the status of the pod. You can analyze the error based on the log data.

Method 3: Print the log of kubelet

If no PV is mounted to a pod, query the IP address of the node that hosts the pod.

kubectl describe pod nginx-97dc96f7b-xbx8t | grep Node

Expected output:

Node: cn-hangzhou.i-bp19myla3uvnt6zi****/192.168.XX.XX
Node-Selectors:  <none>

Log on to the node and print the log of kubelet.

ssh 192.168.XX.XX
journalctl -u kubelet -r -n 1000 &> kubelet.log

Note In the preceding command, -n specifies the number of log entries to return.

The preceding steps describe how to print the logs of the FlexVolume, provisioner, and kubelet plug-ins. If the problem persists, contact the technical support team of Alibaba Cloud and provide related log information.

What do I do if I find kubelet contains a pod log that is not managed by ACK?

When a pod exceptionally exits, the mount target is not removed when the system unmounts the PV. As a result, the system fails to delete the pod. kubelet cannot collect all volume garbage. You must manually remove invalid mount targets or run a script to automate garbage collection.

Run the following script on the failed node to remove invalid mount targets:

wget https://raw.githubusercontent.com/AliyunContainerService/kubernetes-issues-solution/master/kubelet/kubelet.sh
sh kubelet.sh