Container Service for Kubernetes (ACK) allows you to monitor disks and Apsara File Storage NAS (NAS) file systems at the node side by using Prometheus Service. This topic describes how to monitor disks and NAS file systems at the node side and provides an example on how to configure alert rules.

Prerequisites

Only csi-plugin of v1.18.8.46-afb19e46-aliyun can be used to monitor both disks and NAS file systems. Make sure that the YAML file and the image of csi-plugin are upgraded to the latest version. You can upgrade csi-plugin to the latest version by using the one of the following methods:

Background information

In most cases, ACK clusters store data in disks or NAS file systems. You can install the latest version of csi-plugin in ACK clusters to map the drive letters of disks to the persistent volume claims (PVCs) that are used to mount the disks. You can use Prometheus Service to monitor the status of PVCs. You can also configure threshold-based alert rules to monitor the storage usage and input/output operations per second (IOPS). If the specified thresholds are reached, alerts are triggered.

Upgrade csi-plugin by using the ACK console

  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click the name of a cluster and choose Operations > Add-ons in the left-side navigation pane.
  3. Click the Storage tab, find csi-plugin and click Upgrade.

How to monitor storage resources

Only disks and NAS file systems can be monitored at the node side. To view the monitoring data that is provided by Prometheus Service, perform the following steps:

Note Before you monitor NAS file systems, you must enable Prometheus Monitoring. For more information, see Enable ARMS Prometheus.
  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click the name of a cluster and choose Operations > Prometheus Monitoring in the left-side navigation pane.
  3. Click the dashboard named Storage Monitoring, and then click CSI - Nodes in the dashboard that appears, you can view the monitoring data that is collected by csi-plugin from nodes.
    metrics
  • Filters
    Filter Description
    StorageType The type of storage resource. A value of disk indicates disks. A value of nas indicates NAS file systems.
    Namespace The namespace to which the PVC belongs
    PVC The name of the PVC
  • Metrics
    Metric Description Unit
    IOPS The read/write IOPS of the PVC ops/s (number of reads/writes per second)
    Latency(avg) The average latency at which the PVC processes read/write requests ms (milliseconds)
    ThroughPut The throughput of the PVC GB
    IO Count In Queue The number of I/O requests in the queue Count
    Used Capacity The used storage of the PVC GB
    Total Capacity The storage capacity of the PVC GB
    Free Capacity The available storage of the PVC GB

Examples of monitoring disks

  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click the name of a cluster and choose Operations > Prometheus Monitoring in the left-side navigation pane.
  3. On the Prometheus Monitoring page, click Storage Monitoring, and then click CSI - Nodes tab.
  4. Set StorageType and Namespace based on the disk volume that you want to monitor, and select the PVC that is used to mount the disk volume. Then, you can view the capacity of the disk volume in the Total Capacity chart.
    In this example, set StorageType to disk, Namespace to default, and Pvc to pvc-disk-static-create1. Monitor a diskThe preceding figure shows that the total capacity of the disk is 39.25 GiB. 48.2 MiB has been used and 39.18 GiB is available.
  5. Write data into the disk.
    1. Run the following command to query the name of the application to which the disk is mounted: 
      kubectl get pod

      Expected output:

      NAME                                              READY   STATUS    RESTARTS   AGE
deployment-disk-static-create1-67f4b5cfdc-p5q7t   1/1     Running   0          20m
    2. Run the following command to log on to the application: 
      kubectl exec deployment-disk-static-create1-67f4b5cfdc-p5q7t -ti sh
    3. Run the following command to write 20 GiB of data into the /data directory where the disk is mounted: 
      cd /data
/usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
  6. Go to the Prometheus Monitoring page, click Storage Monitoring, and then click CSI - Nodes to check the monitoring data of the disk.
    After data is written into the diskThe preceding figure shows that the write IOPS is 4,000, the throughput is 15 MB/s, the size of the data written into the disk is 20 GiB, the available space is 19 GiB, and the average latency is 67 ms.

Examples of monitoring NAS file systems

  1. Write data into the directory to which the NAS volume is mounted.
    1. Run the following command to query the name of the application to which the NAS volume is mounted: 
      kubectl get pod

      Expected output:

      NAME                                   READY    STATUS    RESTARTS   AGE
cnfs-nas-deployment-84f6cdf6cc-wvjjq   1/1      Running   0          20m
    2. Run the following command to log on to the application: 
      kubectl exec cnfs-nas-deployment-84f6cdf6cc-wvjjq -ti sh
    3. Run the following command to write 20 GiB of data into the /data directory where the NAS volume is mounted: 
      cd /data
/usr/bin/fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -runtime=100 -group_reporting -name=Rand_Write_Testing -filename=./test -size=20G
  2. Log on to the ACK console and click Clusters in the left-side navigation pane.
  3. On the Clusters page, click the name of a cluster and choose Operations > Prometheus Monitoring in the left-side navigation pane.
  4. On the Prometheus Monitoring page, click Storage Monitoring, and then click CSI - Nodes tab.
  5. Set StorageType and Namespace based on the NAS volume, and select the PVC that is used to mount NAS volume. Then, you can view the capacity of the NAS volume in the Total Capacity chart.
    In this example, set StorageType to nas, Namespace to default, and Pvc to cnfs-nas-pvc. Monitor a NAS file systemThe preceding figure shows that the write IOPS is approximately 15,000, the throughput increases from 10 MB/s to 60 MB/s, and the average latency is 15 ms.