Container Service for Kubernetes:High-availability configuration for disk volumes

Configuration goals

• For StatefulSets, we recommend using disks for persistent storage. Compared to NAS, disks offer higher stability and better data transfer bandwidth.

• Ensure the cluster spans multiple availability zones with sufficient compute and storage resources to meet demand.

• The cluster can automatically scale out nodes to meet scheduling demand when an entire availability zone becomes unavailable.

• Use a highly available StorageClass to prevent disk mount failures.

• Ensure that pods are evenly distributed across nodes and availability zones.

Node pool configurations

• Use a single availability zone for each node pool.

  • When you add an availability zone, create a new node pool for it. For more information, see Create and manage a node pool.

  • When you create node pools, ensure that each node pool maps to a different availability zone. Use node pool names to distinguish between the availability zones.

• Enable auto scaling for the node pools. For more information, see Enable node auto scaling.

  If no nodes are available in an availability zone, the system automatically creates a new node in that zone to schedule the pod, as shown in the following log.

  Warning  FailedScheduling     3m26s                xxx                0/2 nodes are available: 2 node(s) had volume node affinity conflict.
  Warning  FailedScheduling     3m26s                xxx                0/2 nodes are available: 2 node(s) had volume node affinity conflict.
  Normal   Scheduled            14s                  xxx                Successfully assigned default/web-csi-available-test4-1 to cn-wulanchabu.172.xxx
  Warning  FailedScheduling     86s                  xxx                0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
  Warning  FailedScheduling     76s                  xxx                0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict.
  Warning  FailedScheduling     25s                  xxx                0/3 nodes are available: 3 node(s) had volume node affinity conflict.
  Normal   TriggeredScaleUp     3m8s                 cluster-autoscaler   pod triggered scale-up: [xxx 0->1 (max: 10)}]
  Normal   NotTriggerScaleUp    2m39s                cluster-autoscaler   pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 can't increase node group size
  Normal   TriggeredScaleUp     37s                  cluster-autoscaler   pod triggered scale-up: [{xxx 1->2 (max: 10)}]
  Normal   SuccessfulAttachVolume 15s                attachdetach-controller  AttachVolume.Attach succeeded for                         i3uqre"
  Warning  FailedMount          9s (x3 over 11s)     kubelet              MountVolume.MountDevice failed for volume "xxx" : rpc error: code = Aborted desc = NodeStageVolume: Attach volume: d-xxx
           with error: rpc error: code = Aborted desc = NodeStageVolume: Previous attach action is still in process: d xxx
  Normal   Pulling              3s                   kubelet              Pulling image "nginx"

• Use the same ECS instance type across all availability zones. If this is not possible, use instance types that all support the same block storage categories.

  Some disk types cannot be mounted on certain ECS instance types. Even if a pod is scheduled within the correct availability zone, it can fail to start due to disk mount errors if the node does not support the requested disk type.

• Configure taints for all node pools to prevent unintended workloads from being scheduled on these nodes and impacting your application. When you create a node pool, you can configure taints for its nodes in the Taints section. For example, set the key to app, the value to sts, and the effect to NoSchedule. This prevents pods without a matching toleration from being scheduled on nodes in this node pool.

Cluster configurations

• Ensure that your cluster version is 1.20 or later.

• Ensure that your CSI plug-in version is 1.22 or later. For more information, see Manage the CSI plug-in.

• Use a highly available StorageClass.

  The following example shows the YAML configuration:

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: alicloud-disk-topology-alltype
  parameters:
    type: cloud_essd,cloud_ssd,cloud_efficiency
  provisioner: diskplugin.csi.alibabacloud.com
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer
  allowVolumeExpansion: true
  allowedTopologies:
  - matchLabelExpressions:
    - key: topology.diskplugin.csi.alibabacloud.com/zone
      values:
      - cn-beijing-a
      - cn-beijing-b

  Key parameters:

  • type: cloud_essd,cloud_ssd,cloud_efficiency: Specifies a prioritized list of disk types for a fallback mechanism. The CSI plug-in attempts to create a disk by trying the specified types in order: ESSD, standard SSD, and then ultra disk. This approach prevents pod startup failures caused by insufficient disk inventory for a specific type.

  • volumeBindingMode: WaitForFirstConsumer: Delays disk creation until a pod is scheduled to a specific node. This ensures the disk is created in the same availability zone as the node, which prevents pod startup failures caused by zone mismatches.

  • allowedTopologies: Restricts volume provisioning to specific availability zones. When volumeBindingMode is set to WaitForFirstConsumer, the scheduler places pods in one of the specified availability zones, ensuring the disk can be provisioned successfully.

Application configurations

The following template shows a standard StatefulSet configuration. You can customize it as needed.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: "mysql"
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            app: mysql
        maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "mysql"
        volumeMounts:
        - name: disk-csi
          mountPath: /var/lib/mysql
      tolerations:
      - key: "app"
        operator: "Exists"
        effect: "NoSchedule"
  volumeClaimTemplates:
  - metadata:
      name: disk-csi
    spec:
      accessModes: [ "ReadWriteMany" ]
      storageClassName: alicloud-disk-topology-alltype
      resources:
        requests:
          storage: 40Gi

Key parameters:

• topologySpreadConstraints: Spreads pods across different availability zones as evenly as possible. For more information, see Topology Spread Constraints.

• volumeClaimTemplates: Automatically creates the required number of disks based on the replica count, which simplifies scaling.