Best practice for throttling block storage devices - Container Service for Kubernetes

Open source Kubernetes does not support throttling of block storage devices. Container Service for Kubernetes (ACK) lets you throttle a block storage device that is mounted to a pod, such as a cloud disk or a local disk that is virtualized using Logical Volume Manager (LVM). This helps you more efficiently manage and allocate storage resources, ensures system performance and stability, and reduces costs. This topic describes how to throttle a block storage device.

Limits

The OS used by the host must be Alibaba Cloud Linux 2 or later.
The Kubernetes version of your ACK cluster must be 1.20 or later.
The version of the Container Storage Interface (CSI) plug-in used by your cluster must be 1.22 or later. For more information about CSI versions, see csi-provisioner.

Parameters

readIOPS: specifies the maximum number of read operations per second that a pod can perform on a block storage device.
writeIOPS: specifies the maximum number of write operations per second that a pod can perform on a block storage device.
readBPS: specifies the maximum amount of data per second that a pod can read from a block storage device.
writeBPS: specifies the maximum amount of data per second that a pod can write to a block storage device.

Usage

The following example shows how to throttle a cloud disk. The cloud disk can be mounted as a dynamically provisioned volume or a statically provisioned volume.

Note

After throttling settings are configured for a cloud disk, they cannot be changed. If you adjust the throttling configuration in a StorageClass, the changes apply only to newly created cloud disks. Existing cloud disks are not affected.

Mount the disk as a dynamically provisioned volume

Create a file named alicloud-disk-topology-essd.yaml and copy the following content to the file:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-disk-topology-essd
parameters:
  type: cloud_essd
  readIOPS: "100"
  writeIOPS: "10"
  readBPS: "100k"
  writeBPS: "100m"
provisioner: diskplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Run the following command to create a StorageClass:
```
kubectl apply -f alicloud-disk-topology-essd.yaml
```

Create a file named nginx.yaml and copy the following content to the file:

The application uses a persistent volume claim (PVC) that references the StorageClass you created to dynamically provision a cloud disk as a volume.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-csi-encrypt
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  podManagementPolicy: "Parallel"
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      containers:
      - name: nginx
        command:
        - sleep
        - "999999999"
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        volumeMounts:
        - name: disk-csi
          mountPath: /data
  volumeClaimTemplates:
  - metadata:
      name: disk-csi
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: alicloud-disk-topology-essd
      resources:
        requests:
          storage: 80Gi

Run the following command to create the application:
```
kubectl apply -f nginx.yaml
```

View the block device limits for the container based on the cgroup version.

Run the following command to check the cgroup version:
```
stat -fc %T /sys/fs/cgroup
```
If the output is cgroup2fs, the version is cgroup V2. Otherwise, it is cgroup V1.

View the block device limits for the container based on the cgroup version.

cgroup V2

You need to log on to the host and use the pod UID to construct the path. Run the following command to verify that the limits are set correctly:

/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod<UID>.slice/io.max

cgroup V1

You need to log on to the host and use the pod UID to construct the path. Run the following commands to verify that the limits are set correctly:

readIOPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.read_iops_device

writeIOPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.write_iops_device

readBPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.read_bps_device

writeBPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.write_bps_device

Mount the disk as a statically provisioned volume

Use the following template to create a file named pv-static.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: d-****
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 80Gi
  csi:
    driver: diskplugin.csi.alibabacloud.com
    fsType: ext4
    volumeAttributes:
      app: nginx
      type: cloud_ssd
      readBPS: 100K
      readIOPS: "100"
    volumeHandle: d-****
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

Run the following command to create a PV:
```
kubectl apply -f pv-static.yaml
```

Use the following template to create a pvc-static.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: nginx
  name: disk-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 80Gi
  volumeMode: Filesystem
  volumeName: d-****

Run the following command to create a PVC:
```
kubectl apply -f pvc-static.yaml
```

Create a file named nginx.yaml and copy the following content to the file:

The application uses the PVC that you created. The PVC is used to statically provision a cloud disk as a volume.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploy-disk
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        volumeMounts:
        - name: disk-pvc2
          mountPath: /data
      volumes:
        - name: disk-pvc2
          persistentVolumeClaim:
            claimName: disk-pvc

Run the following command to create the application:
```
kubectl apply -f nginx.yaml
```

View the block device limits for the container based on the cgroup version.

Run the following command to check the cgroup version:
```
stat -fc %T /sys/fs/cgroup
```
If the output is cgroup2fs, the version is cgroup V2. Otherwise, it is cgroup V1.

View the block device limits for the container based on the cgroup version.

cgroup V2

You need to log on to the host and use the pod UID to construct the path. Run the following command to verify that the limits are set correctly:

/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod<UID>.slice/io.max

cgroup V1

You need to log on to the host and use the pod UID to construct the path. Run the following commands to verify that the limits are set correctly:

readIOPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.read_iops_device

writeIOPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.write_iops_device

readBPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.read_bps_device

writeBPS:

/sys/fs/cgroup/blkio/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod{pod_uid}.slice/blkio.throttle.write_bps_device