Enable disk parallel mounting - Container Service for Kubernetes

When many pods share a node and each requires multiple disks—common in database, continuous integration (CI), and batch processing workloads—the default serial mounting increases pod startup time. Disk parallel mounting attaches disks concurrently, reducing startup latency.

Prerequisites

You have an ACK managed cluster running Kubernetes 1.26 or later, with csi-plugin and csi-provisioner both at v1.30.4 or later.
You have installed and configured Alibaba Cloud CLI.

Usage notes

Parallel mounting is supported only for disks that have serial numbers. For more information about how to query the serial number of a disk, see Query serial number.
Disks created before June 10, 2020 do not have recognizable serial numbers. If you enable parallel mounting for these disks, the disks fail to mount.
When multiple disks are unmounted from the same node, the unmount operations still run in serial mode.
After you enable parallel mounting, the Device fields returned by the API, such as ECS DescribeDisks, and the mount target displayed in the console may be inaccurate. Do not rely on this mount path in your workloads. Use the serial number of the disk to determine the actual mount path.

Procedure

To enable disk parallel mounting, use either the automated script or the manual procedure below.

Use an automated script

Save the following script as a file named enable_parallel_attach.sh.

View the script

#!/bin/bash

set -e
set -o pipefail

readonly REQUIRED_VERSION="v1.30.4"
CLUSTER_ID=$1

if [ -z "$CLUSTER_ID" ]; then
    echo "Usage: enable_parallel_attach.sh <cluster-id>"
    exit 1
fi

check_version() {
    local ADDONS VERSION
    ADDONS=$(aliyun cs GET "/clusters/${CLUSTER_ID}/addon_instances")

    VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="csi-plugin") | .version')
    if !  printf "%s\n" "$REQUIRED_VERSION" "$VERSION" | sort -V -C; then
        echo "csi-plugin version $VERSION is not supported, please upgrade to $REQUIRED_VERSION or later"
        exit 1
    fi

    PROVISIONER=managed-csiprovisioner
    VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="managed-csiprovisioner") | .version')
    if [ -z "$VERSION" ]; then
        PROVISIONER=csi-provisioner
        VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="csi-provisioner") | .version')
    fi
    if !  printf "%s\n" "$REQUIRED_VERSION" "$VERSION" | sort -V -C; then
        echo "$PROVISIONER version $VERSION is not supported, please upgrade to $REQUIRED_VERSION or later"
        exit 1
    fi
}

update_node_pool() {
    local NODE_POOL_DOC
    NODE_POOL_DOC=$(aliyun cs GET "/clusters/${CLUSTER_ID}/nodepools/$1")

    if [ -n "$(echo "$NODE_POOL_DOC" | jq -r '(.scaling_group.tags // [])[] | select(.key=="supportConcurrencyAttach")')" ]; then
        echo "node pool already has supportConcurrencyAttach tag"
        return
    fi

    aliyun cs PUT "/clusters/${CLUSTER_ID}/nodepools/$1" --header "Content-Type=application/json" \
        --body "$(echo "$NODE_POOL_DOC" | jq -c '{
    "scaling_group": {
        "tags": ((.scaling_group.tags // []) + [{
            "key": "supportConcurrencyAttach",
            "value": "true"
        }])
    }
}')"
}

# Configure existing nodes.
update_nodes() {
    local PAGE=1
    local IDX TOTAL NODES_DOC ARGS
    while :; do
        echo "tagging nodes, page $PAGE"
        NODES_DOC=$(aliyun cs GET "/clusters/${CLUSTER_ID}/nodes" --pageSize 50 --pageNumber $PAGE)
        TOTAL=$(echo "$NODES_DOC" | jq -r '.page.total_count')

        ARGS=()
        IDX=0
        for node in $(echo "$NODES_DOC" | jq -r '.nodes[] | select(.is_aliyun_node) | .instance_id'); do
            IDX=$((IDX+1))
            ARGS+=("--ResourceId.$IDX" "$node")
        done
        if [ "$IDX" != "0" ]; then
            aliyun ecs TagResources --region "$ALIBABA_CLOUD_REGION_ID" --ResourceType Instance "${ARGS[@]}" \
                --Tag.1.Key supportConcurrencyAttach --Tag.1.Value true
            echo "finished nodes $(( (PAGE-1)*50+IDX ))/$TOTAL"
        fi

        if [[ $(( PAGE*50 )) -ge $TOTAL ]]; then
            break
        fi
        PAGE=$((PAGE+1))
    done
}

update_addon() {
    local ADDON=$1
    shift
    local CONFIG STATE
    CONFIG=$(aliyun cs GET "/clusters/${CLUSTER_ID}/addon_instances/${ADDON}" | \
        jq -c '.config | fromjson | (.FeatureGate // "" | split(",")) as $fg | .FeatureGate = ($fg + $ARGS.positional | unique | join(",")) | {config: . | tojson}' --args "$@")

    aliyun cs POST "/clusters/${CLUSTER_ID}/components/${ADDON}/config" --header "Content-Type=application/json" --body "$CONFIG"

    echo "Waiting for $ADDON config to complete"
    while true; do
        STATE=$(aliyun --secure cs GET "/clusters/${CLUSTER_ID}/addon_instances/${ADDON}" | jq -r '.state')
        echo "state: $STATE"
        if [ "$STATE" != "updating" ]; then
            break
        fi
        sleep 5
    done
    if [ "$STATE" != "active" ]; then
        echo "Failed to update $ADDON config"
        return 1
    fi
}

check_version

aliyun cs GET "/clusters/${CLUSTER_ID}/nodepools" | jq -r '.nodepools[]|.nodepool_info|"\(.nodepool_id)\t\(.name)"' | \
while read -r NODE_POOL_ID NODE_POOL_NAME; do
    echo "Updating tags for node pool $NODE_POOL_NAME ($NODE_POOL_ID)"
    update_node_pool "$NODE_POOL_ID"
done

ALIBABA_CLOUD_REGION_ID=$(aliyun cs GET "/clusters/${CLUSTER_ID}" | jq -r .region_id)

update_nodes

update_addon $PROVISIONER DiskADController=true DiskParallelAttach=true
update_addon csi-plugin DiskADController=true

echo "All done!  Now the disks can be attached concurrently to the same node."

Execute the script to mount disks in parallel mode.
```
bash enable_parallel_attach.sh <Cluster ID>
```

Manually enable the feature

Add ECS Tags to the node pool of the cluster. Set the tag key to supportConcurrencyAttach and the tag value to true. Make sure that the tag is added to a new ECS instance.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
3. On the Node Pools page, find the node pool that you want to modify and click Edit in the Actions column.
4. In the lower part of the page, find the Advanced Options section and add an ECS Tags. Set the key to supportConcurrencyAttach and the value to true.
Add a tag to the ECS instances of all existing nodes in the cluster. Set the key to supportConcurrencyAttach and the value to true. For more information, see Add a custom tag.
In the left-side navigation pane, click Components and Add-ons. Click the Volumes tab, find the csi-provisioneradd-on, click Configure in the lower-right corner of the component, and set the FeatureGate parameter to DiskADController=true,DiskParallelAttach=true.
Note
After you specify DiskADController=true, the attach and detach operations related to disks are performed by csi-provisioner. After you specify DiskParallelAttach=true, the disk parallel mounting feature is enabled.
After you configure csi-provisioner, set the FeatureGate parameter of csi-plugin to DiskADController=true.

Verify disk parallel mounting performance

In this example, pods with a large number of disks mounted are created on the same node to verify pod startup acceleration after parallel mounting is enabled.

Important

The test results in this topic are reference values. Actual results may vary depending on your environment.

Add a node that supports multiple disks to an ACK cluster. For example, an ecs.g7se.16xlarge instance supports up to 56 disks.

Create a test file named attach-stress.yaml and copy the following content to the file. Replace <YOUR-HOSTNAME>with the node's actual name.

View the attach-stress.yaml file

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alibabacloud-disk
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_auto
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: attach-stress
spec:
  selector:
    matchLabels:
      app: attach-stress
  serviceName: attach-stress
  replicas: 1
  podManagementPolicy: Parallel
  persistentVolumeClaimRetentionPolicy:
    whenScaled: Retain
    whenDeleted: Delete
  template:
    metadata:
      labels:
        app: attach-stress
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - <YOUR-HOSTNAME> # Replace this parameter with the actual name of the node. 
      hostNetwork: true
      containers:
      - name: attach-stress
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/busybox
        command: ["/bin/sh", "-c", "trap exit TERM; while true; do date > /mnt/0/data; sleep 1; done"]
        volumeMounts:
        - name: volume-0
          mountPath: /mnt/0
        - name: volume-1
          mountPath: /mnt/1
  volumeClaimTemplates:
  - metadata:
      name: volume-0
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alibabacloud-disk
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: volume-1
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alibabacloud-disk
      resources:
        requests:
          storage: 1Gi

Run the following command to confirm that the application starts as expected. Then, scale the replica count to 0 to prepare for the batch mount tests.

kubectl apply -f attach-stress.yaml
kubectl rollout status sts attach-stress
kubectl scale sts attach-stress --replicas 0

Expected output:

storageclass.storage.k8s.io/alibabacloud-disk created
statefulset.apps/attach-stress created
partitioned roll out complete: 1 new pods have been updated...
statefulset.apps/attach-stress scaled

Run the following commands to start the batch mount test and measure the pod startup time:

Note

In this case, parallel mounting is disabled for the cluster. Adjust the number of pods for the test based on the maximum number of disks supported by your node.

date && \
  kubectl scale sts attach-stress --replicas 28 && \
  kubectl rollout status sts attach-stress && \
  date

Expected output:

Tuesday October 15 19:21:36 CST 2024
statefulset.apps/attach-stress scaled
Waiting for 28 pods to be ready...
Waiting for 27 pods to be ready...
<Omitted...>
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 28 new pods have been updated...
Tuesday October 15 19:24:55 CST 2024

The output indicates that more than 3 minutes is required for starting all 28 pods when parallel mounting is disabled.

Enable parallel mounting by following the instructions in the Procedure section.
Run the following command to delete the preceding pods and prepare for the subsequent round of testing:
Note
Monitor the volumeattachments resources in the cluster. After these resources are deleted, the disks are unmounted. This process takes a few minutes.
```
kubectl scale sts attach-stress --replicas 0
```

Run the following command again to calculate the time required for starting pods after parallel mounting is enabled. The expected time is approximately 40 seconds, which is much faster than the 3 minutes when parallel mounting is disabled.

date && \
  kubectl scale sts attach-stress --replicas 28 && \
  kubectl rollout status sts attach-stress && \
  date

Expected output:

Tuesday October 15 20:02:54 CST 2024
statefulset.apps/attach-stress scaled
Waiting for 28 pods to be ready...
Waiting for 27 pods to be ready...
<Omitted...>
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 28 new pods have been updated...
Tuesday October 15 20:03:31 CST 2024

Run the following command to delete the test applications in the cluster:
```
kubectl delete -f attach-stress.yaml
```