Enable disk parallel mounting - Container Service for Kubernetes

In scenarios that involve high-density deployments of stateful applications (such as databases) or large numbers of short-lived containers (such as continuous integration and batch processing), each pod requires a large number of disks to persist data. When a large number of pods are scheduled to the same node at the same time, the default serial mounting method increases the pod startup time. To resolve this issue, you can enable the disk parallel mounting feature.

Prerequisites

An ACK managed cluster is created. The Kubernetes version of the cluster is 1.26 or later. The versions of the csi-plugin and csi-provisioner components are 1.30.4 or later. For more information, see csi-plugin and csi-provisioner.
Alibaba Cloud CLI is installed and configured. For more information, see Install Alibaba Cloud CLI.

Usage notes

You can enable the parallel mounting feature only for disks that have serial numbers. For more information about how to query the serial number of a disk, see Query the serial numbers of block storage devices.
Disks that were created before June 10, 2020 do not have recognizable serial numbers. If you enable the parallel mounting feature for these disks, mounting failures will occur.
When multiple disks are unmounted from the same node, the disks are unmounted in serial mode.
After you enable parallel mounting, the Device fields returned by the API, such as ECS DescribeDisks, and the mount target displayed in the console may be inaccurate. Do not use this mount path in your business. You can use the serial number of the disk to confirm the actual mount path.

Procedure

You can manually enable disk parallel mounting or use an automated script to enable this feature.

Use an automated script

Save the following script as a file named enable_parallel_attach.sh.

View the script

#!/bin/bash

set -e
set -o pipefail

readonly REQUIRED_VERSION="v1.30.4"
CLUSTER_ID=$1

if [ -z "$CLUSTER_ID" ]; then
    echo "Usage: enable_parallel_attach.sh <cluster-id>"
    exit 1
fi

check_version() {
    local ADDONS VERSION
    ADDONS=$(aliyun cs GET "/clusters/${CLUSTER_ID}/addon_instances")

    VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="csi-plugin") | .version')
    if !  printf "%s\n" "$REQUIRED_VERSION" "$VERSION" | sort -V -C; then
        echo "csi-plugin version $VERSION is not supported, please upgrade to $REQUIRED_VERSION or later"
        exit 1
    fi

    PROVISIONER=managed-csiprovisioner
    VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="managed-csiprovisioner") | .version')
    if [ -z "$VERSION" ]; then
        PROVISIONER=csi-provisioner
        VERSION=$(echo "$ADDONS" | jq -r '.addons[] | select(.name=="csi-provisioner") | .version')
    fi
    if !  printf "%s\n" "$REQUIRED_VERSION" "$VERSION" | sort -V -C; then
        echo "$PROVISIONER version $VERSION is not supported, please upgrade to $REQUIRED_VERSION or later"
        exit 1
    fi
}

update_node_pool() {
    local NODE_POOL_DOC
    NODE_POOL_DOC=$(aliyun cs GET "/clusters/${CLUSTER_ID}/nodepools/$1")

    if [ -n "$(echo "$NODE_POOL_DOC" | jq -r '(.scaling_group.tags // [])[] | select(.key=="supportConcurrencyAttach")')" ]; then
        echo "node pool already has supportConcurrencyAttach tag"
        return
    fi

    aliyun cs PUT "/clusters/${CLUSTER_ID}/nodepools/$1" --header "Content-Type=application/json" \
        --body "$(echo "$NODE_POOL_DOC" | jq -c '{
    "scaling_group": {
        "tags": ((.scaling_group.tags // []) + [{
            "key": "supportConcurrencyAttach",
            "value": "true"
        }])
    }
}')"
}

# Configure existing nodes.
update_nodes() {
    local PAGE=1
    local IDX TOTAL NODES_DOC ARGS
    while :; do
        echo "tagging nodes, page $PAGE"
        NODES_DOC=$(aliyun cs GET "/clusters/${CLUSTER_ID}/nodes" --pageSize 50 --pageNumber $PAGE)
        TOTAL=$(echo "$NODES_DOC" | jq -r '.page.total_count')

        ARGS=()
        IDX=0
        for node in $(echo "$NODES_DOC" | jq -r '.nodes[] | select(.is_aliyun_node) | .instance_id'); do
            IDX=$((IDX+1))
            ARGS+=("--ResourceId.$IDX" "$node")
        done
        if [ "$IDX" != "0" ]; then
            aliyun ecs TagResources --region "$ALIBABA_CLOUD_REGION_ID" --ResourceType Instance "${ARGS[@]}" \
                --Tag.1.Key supportConcurrencyAttach --Tag.1.Value true
            echo "finished nodes $(( (PAGE-1)*50+IDX ))/$TOTAL"
        fi

        if [[ $(( PAGE*50 )) -ge $TOTAL ]]; then
            break
        fi
        PAGE=$((PAGE+1))
    done
}

update_addon() {
    local ADDON=$1
    shift
    local CONFIG STATE
    CONFIG=$(aliyun cs GET "/clusters/${CLUSTER_ID}/addon_instances/${ADDON}" | \
        jq -c '.config | fromjson | (.FeatureGate // "" | split(",")) as $fg | .FeatureGate = ($fg + $ARGS.positional | unique | join(",")) | {config: . | tojson}' --args "$@")

    aliyun cs POST "/clusters/${CLUSTER_ID}/components/${ADDON}/config" --header "Content-Type=application/json" --body "$CONFIG"

    echo "Waiting for $ADDON config to complete"
    while true; do
        STATE=$(aliyun --secure cs GET "/clusters/${CLUSTER_ID}/addon_instances/${ADDON}" | jq -r '.state')
        echo "state: $STATE"
        if [ "$STATE" != "updating" ]; then
            break
        fi
        sleep 5
    done
    if [ "$STATE" != "active" ]; then
        echo "Failed to update $ADDON config"
        return 1
    fi
}

check_version

aliyun cs GET "/clusters/${CLUSTER_ID}/nodepools" | jq -r '.nodepools[]|.nodepool_info|"\(.nodepool_id)\t\(.name)"' | \
while read -r NODE_POOL_ID NODE_POOL_NAME; do
    echo "Updating tags for node pool $NODE_POOL_NAME ($NODE_POOL_ID)"
    update_node_pool "$NODE_POOL_ID"
done

ALIBABA_CLOUD_REGION_ID=$(aliyun cs GET "/clusters/${CLUSTER_ID}" | jq -r .region_id)

update_nodes

update_addon $PROVISIONER DiskADController=true DiskParallelAttach=true
update_addon csi-plugin DiskADController=true

echo "All done!  Now the disks can be attached concurrently to the same node."

Execute the script to mount disks in parallel mode.
```
bash enable_parallel_attach.sh <Cluster ID>
```

Manually enable the feature

Add an ECS tag to the node pool of the cluster. Set the tag key to supportConcurrencyAttach and the tag value to true. Make sure that the tag is added to a new ECS instance.
1. Log on to the ACK console. In the left-side navigation pane, click Clusters.
2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.
3. On the Node Pools page, find the node pool that you want to modify and click Edit in the Actions column.
4. In the lower part of the page, find the Advanced Options section and add an ECS tag. Set the key to supportConcurrencyAttach and the value to true.
Add a tag to the ECS instances of all existing nodes in the cluster. Set the key to supportConcurrencyAttach and the value to true. For more information, see Add a custom tag.
In the left-side navigation pane, choose Operations > Add-ons. Click the Storage tab, find the csi-provisioner component, click Configure in the lower-right corner of the component, and set the FeatureGate parameter to DiskADController=true,DiskParallelAttach=true.
Note
After you specify DiskADController=true, the attach and detach operations related to disks are performed by csi-provisioner. After you specify DiskParallelAttach=true, the disk parallel mounting feature is enabled.
After you configure csi-provisioner, set the FeatureGate parameter of the csi-plugin component to DiskADController=true.

Verify that the disk parallel mounting feature is enabled

In this example, pods with a large number of disks mounted are created on the same node to verify pod startup acceleration after parallel mounting is enabled.

Important

The statistics provided in this topic are only theoretical values. The actual values may vary based on your environment.

Add a node that supports multiple disks to an ACK cluster. For example, you can mount up to 56 disks to an instance of the ecs.g7se.16xlarge type.

Create a test file named attach-stress.yaml and copy the following content to the file. Replace attach-stress.yaml with the actual name of the node.

View the attach-stress.yaml file

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alibabacloud-disk
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_auto
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: attach-stress
spec:
  selector:
    matchLabels:
      app: attach-stress
  serviceName: attach-stress
  replicas: 1
  podManagementPolicy: Parallel
  persistentVolumeClaimRetentionPolicy:
    whenScaled: Retain
    whenDeleted: Delete
  template:
    metadata:
      labels:
        app: attach-stress
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - <YOUR-HOSTNAME> # Replace this parameter with the actual name of the node. 
      hostNetwork: true
      containers:
      - name: attach-stress
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/busybox
        command: ["/bin/sh", "-c", "trap exit TERM; while true; do date > /mnt/0/data; sleep 1; done"]
        volumeMounts:
        - name: volume-0
          mountPath: /mnt/0
        - name: volume-1
          mountPath: /mnt/1
  volumeClaimTemplates:
  - metadata:
      name: volume-0
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alibabacloud-disk
      resources:
        requests:
          storage: 1Gi
  - metadata:
      name: volume-1
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alibabacloud-disk
      resources:
        requests:
          storage: 1Gi

Run the following command to confirm that the application starts as expected. Then, scale in the number of pods to 0 to prepare for the subsequent batch mount tests.

kubectl apply -f attach-stress.yaml
kubectl rollout status sts attach-stress
kubectl scale sts attach-stress --replicas 0

Expected output:

storageclass.storage.k8s.io/alibabacloud-disk created
statefulset.apps/attach-stress created
partitioned roll out complete: 1 new pods have been updated...
statefulset.apps/attach-stress scaled

Run the following command to start the batch mount test and calculate the time required for starting pods:

Note

In this case, parallel mounting is disabled for the cluster. Adjust the number of pods for the test based on the maximum number of disks supported by your node.

date && \
  kubectl scale sts attach-stress --replicas 28 && \
  kubectl rollout status sts attach-stress && \
  date

Expected output:

Tuesday October 15 19:21:36 CST 2024
statefulset.apps/attach-stress scaled
Waiting for 28 pods to be ready...
Waiting for 27 pods to be ready...
<Omitted...>
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 28 new pods have been updated...
Tuesday October 15 19:24:55 CST 2024

The output indicates that more than 3 minutes is required for starting all 28 pods when parallel mounting is disabled.

Enable parallel mounting by following the instructions in the Procedure section.
Run the following command to delete the preceding pods and prepare for the subsequent round of testing:
Note
Pay attention to the volumeattachments resources in the cluster. After the resources are deleted, the disks are unmounted. This process requires a few minutes.
```
kubectl scale sts attach-stress --replicas 0
```

Run the following command again to calculate the time required for starting pods after parallel mounting is enabled. The expected time is approximately 40 seconds, which is much faster than the 3 minutes when parallel mounting is disabled.

date && \
  kubectl scale sts attach-stress --replicas 28 && \
  kubectl rollout status sts attach-stress && \
  date

Expected output:

Tuesday October 15 20:02 54 CST 2024
statefulset.apps/attach-stress scaled
Waiting for 28 pods to be ready...
Waiting for 27 pods to be ready...
<Omitted...>
Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 28 new pods have been updated...
Tuesday October 15 20:03:31 CST 2024

Run the following command to delete the test applications in the cluster:
```
kubectl delete -f attach-stress.yaml
```