All Products
Search
Document Center

Container Service for Kubernetes:Use csi-plugin to monitor storage resources on the node side

Last Updated:Dec 19, 2023

The storage monitoring feature of Container Service for Kubernetes (ACK) uses Managed Service for Kubernetes to monitor the usage of disks and Apsara File Storage NAS (NAS) file systems on the node side. This topic describes how to monitor disks and NAS file systems on the node side and provides an example on how to configure alert rules.

Prerequisites

Only csi-plugin v1.18.8.46-afb19e46-aliyun supports both disk monitoring and NAS file system monitoring. Make sure that the YAML file and image of csi-plugin are updated to the latest version. You can update csi-plugin to the latest version by using the one of the following methods:

Background information

Disk volumes or NAS volumes are commonly used to store data in ACK clusters. By default, the latest version of csi-plugin is automatically installed in ACK clusters to monitor persistent volume claims (PVCs) with the storage monitoring feature. Managed Service for Prometheus is used to monitor the status of PVCs and configure the capacity threshold and IOPS threshold. This way, alerts can be generated when the usage of the PVC is abnormal.

Update csi-plugin by using the ACK console

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Add-ons in the left-side navigation pane.

  3. Click the Storage tab, find csi-plugin and click Upgrade.

How to monitor storage resources

Only disks and NAS file systems can be monitored on the node side. To view the monitoring data that is provided by Managed Service for Prometheus, perform the following steps:

Important
  • To view storage resource monitoring data, you need to first enable Managed Service for Prometheus. For more information, see Enable Managed Service for Prometheus.

  • After the storage monitoring feature is enabled, the component will automatically report metrics to Managed Service for Prometheus. These metrics are considered custom metrics. You may need to pay for custom metrics. We recommend that you read and understand the Billing overview topic of Managed Service for Prometheus before you enable the storage monitoring feature. The fee varies based on the size of your cluster and the number of applications deployed in the cluster. You can use the resource usage feature to monitor and manage the usage of your resources.

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Prometheus Monitoring in the left-side navigation pane.

  3. Click the Storage Monitoring tab and then click the CSI - Nodes tab to view the node dashboard.

    metrics

  • Filters

    Filter

    Description

    StorageType

    The type of storage resource. A value of disk indicates disks. A value of nas indicates NAS file systems.

    Namespace

    The namespace to which the PVC belongs

    PVC

    The name of the PVC.

  • Metrics

    Metric

    Description

    Unit

    IOPS

    The read/write IOPS of the PVC

    ops/s (number of reads/writes per second)

    Latency(avg)

    The average latency at which the PVC processes read/write requests

    ms (milliseconds)

    ThroughPut

    The throughput of the PVC

    GB

    IO Count In Queue

    The number of I/O requests in the queue

    Count

    Used Capacity

    The used storage of the PVC

    GB

    Total Capacity

    The storage capacity of the PVC

    GB

    Free Capacity

    The available storage of the PVC

    GB

Examples of disk monitoring

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Prometheus Monitoring in the left-side navigation pane.

  3. On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab.

  4. Specify StorageType, Namespace, and PVC name of the disk volume. Then, you can view the current capacity of the disk volume in the Total Capacity section.

    In this example, set StorageType to disk, Namespace to default, and Pvc to pvc-disk-static-create1. 云盘存储监控资源The preceding figure shows that the total capacity of the disk is 39.25 GiB. 48.2 MiB has been used and 39.18 GiB is available.

  5. Write data into the disk.

    1. Run the following command to query the name of the application to which the disk is mounted:

      kubectl get pod

      Expected output:

      NAME                                              READY   STATUS    RESTARTS   AGE
      deployment-disk-static-create1-67f4b5cfdc-p5q7t   1/1     Running   0          20m
    2. Run the following command to log on to the application:

      kubectl exec deployment-disk-static-create1-67f4b5cfdc-p5q7t -ti sh
    3. Run the following command to write 20 GiB of data into the /data directory where the disk is mounted:

      cd /data
      /usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
  6. On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab to view the monitoring data of the disk volume.

    云盘存储监控数据写入后The preceding figure shows that the write IOPS is 4,000, the throughput is 15 MB/s, 20 GiB of data is written, the remaining disk space is 19 GiB, and the average latency is 67 milliseconds.

Examples of NAS file system monitoring

Important

To use NAS file system monitoring, the persistent volume (PV) from which metrics are collected must meet the following requirements:

  • The NAS file system is managed by Container Network File System (CNFS) and the NAS file system is of the Capacity type.

  • The PV is a dynamically provisioned volume created by using a StorageClass.

  1. Write data into the directory to which the NAS volume is mounted.

    1. Run the following command to query the name of the application to which the NAS volume is mounted:

      kubectl get pod

      Expected output:

      NAME                                   READY    STATUS    RESTARTS   AGE
      cnfs-nas-deployment-84f6cdf6cc-wvjjq   1/1      Running   0          20m
    2. Run the following command to log on to the application:

      kubectl exec cnfs-nas-deployment-84f6cdf6cc-wvjjq -ti sh
    3. Run the following command to write 20 GiB of data into the /data directory where the NAS volume is mounted:

      cd /data
      /usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
  2. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  3. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Prometheus Monitoring in the left-side navigation pane.

  4. On the Prometheus Monitoring page, click the Storage Monitoring tab and then click the CSI - Nodes tab.

  5. Specify StorageType, Namespace, and PVC name of the NAS volume. Then, you can view the current capacity of the NAS volume in the Total Capacity section.

    In this example, set StorageType to nas, Namespace to default, and Pvc to cnfs-nas-pvc. NAS存储监控资源The preceding figure shows that the write IOPS is approximately 15,000, the throughput increases from 10 MB/s to 60 MB/s, and the average latency is 15 milliseconds.