Container storage monitoring - Container Service for Kubernetes

You can use Managed Service for Prometheus to monitor storage resources in a cluster, including the root file system and ephemeral storage, and external storage resources used by the cluster, including disks, File Storage NAS (NAS) file systems, and Object Storage Service (OSS) buckets. The container storage monitoring feature provides storage metrics free of charge to help you monitor storage resources from various perspectives.

Monitor the storage resources in a cluster

The storage resources in a cluster include the root file system and ephemeral storage.

Root file system: stores the data of the container runtime environment, including the data and configurations required by the container runtime. When you use Managed Service for Prometheus to monitor the root file system, Managed Service for Prometheus provides information about the storage usage, utilization, and distribution of the root file system.
Ephemeral storage: stores the temporary data of the container runtime, including the temporary files and cache generated by the container runtime. When you use Managed Service for Prometheus to monitor ephemeral storage, Managed Service for Prometheus provides information about the storage usage, utilization, and distribution of ephemeral storage.

Metrics related to the root file system

Important

If your cluster runs Kubernetes 1.22 or later, you must first update the Container Storage Interface (CSI) plug-in to 1.28.3-eb95171-aliyun or later. For more information about the release notes for csi-plugin, see csi-plugin. When you update the CSI plug-in, you must update both csi-plugin and csi-provisioner.

Metric	Type	Description
container_fs_limit_bytes	Gauge	The total storage capacity of the root file system. Unit: bytes.
container_fs_usage_bytes	Gauge	The storage capacity usage of the root file system. Unit: bytes.
container_fs_available_bytes	Gauge	The available storage capacity of the root file system. Unit: bytes.
container_fs_inodes_total	Gauge	The total number of inodes in the root file system.
container_fs_inodes_used	Gauge	The number of used inodes in the root file system.
container_fs_inodes_free	Gauge	The number of available inodes in the root file system.
container_fs_reads_bytes_total	Gauge	The total size of data that is read from the root file system. Unit: bytes.
container_fs_writes_bytes_total	Gauge	The total size of data that is written to the root file system. Unit: bytes.

Metrics related to ephemeral storage

Important

The following metrics are reliant on csi-plugin. You must install CSI 1.28.3-eb95171-aliyun or later. For more information about the release notes for csi-plugin, see csi-plugin. When you update the CSI plug-in, you must update both csi-plugin and csi-provisioner.

Metric	Type	Description
ephemeral_storage_pod_limit_bytes	Gauge	The total capacity of the ephemeral storage in the pod. Unit: bytes.
ephemeral_storage_pod_usage_bytes	Gauge	The ephemeral storage usage in the pod. Unit: bytes.
ephemeral_storage_pod_available_bytes	Gauge	The available ephemeral storage in the pod. Unit: bytes.
ephemeral_storage_pod_inodes_total	Gauge	The total number of inodes in the ephemeral storage in the pod.
ephemeral_storage_pod_inodes_used	Gauge	The number of used inodes in the ephemeral storage in the pod.
ephemeral_storage_pod_inodes_free	Gauge	The number of available inodes in the ephemeral storage in the pod.

Monitor the external storage resources used by a cluster

Metrics related to disk volumes

Important

The following metrics are reliant on csi-plugin. You must install CSI 1.18.8.46-afb19e46-aliyun or later. For more information about the release notes for csi-plugin, see csi-plugin. When you update the CSI plug-in, you must update both csi-plugin and csi-provisioner.

Metric	Type	Description
node_volume_capacity_bytes_total	Counter	The total capacity of the disk. Unit: bytes.
node_volume_capacity_bytes_used	Counter	The disk capacity usage. Unit: bytes.
node_volume_capacity_bytes_available	Counter	The available capacity of the disk. Unit: bytes.
node_volume_inodes_total	Counter	The total number of inodes on the disk.
node_volume_inodes_used	Counter	The number of used inodes on the disk.
node_volume_inodes_available	Counter	The number of available inodes on the disk.
node_volume_read_bytes_total	Counter	The total size of data that is successfully read from the disk. Unit: bytes.
node_volume_read_completed_total	Counter	The total number of successful read operations.
node_volume_read_merged_total	Counter	The total number of read merge operations performed by the kernel.
node_volume_read_time_milliseconds_total	Counter	The total time consumption of read operations. Unit: milliseconds.
node_volume_write_bytes_total	Counter	The total size of data that is successfully written to the disk. Unit: bytes.
node_volume_write_completed_total	Counter	The total number of successful write operations.
node_volume_write_merged_total	Counter	The total number of write merge operations performed by the kernel.
node_volume_write_time_milliseconds_total	Counter	The total time consumption of write operations. Unit: milliseconds.
node_volume_io_now	Gauge	The number of I/O operations that are in progress.
node_volume_io_time_seconds_total	Counter	The total time consumption of I/O operations. Unit: seconds.

Metrics related to NAS volumes

Important

Metric	Type	Description
node_volume_capacity_bytes_total	Counter	The total capacity of the NAS volume. Unit: bytes.
node_volume_capacity_bytes_used	Counter	The NAS volume capacity usage. Unit: bytes.
node_volume_capacity_bytes_available	Counter	The available capacity of the NAS volume. Unit: bytes.
node_volume_read_bytes_total	Counter	The total size of data that is successfully read from the NAS volume. Unit: bytes.
node_volume_read_sent_bytes_total	Counter	The total size of data that is sent by read requests. Unit: bytes.
node_volume_read_completed_total	Counter	The total number of successful read operations.
node_volume_read_transmissions_total	Counter	The total number of read requests.
node_volume_read_timeouts_total	Counter	The total number of request timeouts.
node_volume_read_time_milliseconds_total	Counter	The total time consumption of read operations. Unit: milliseconds.
node_volume_read_queue_time_milliseconds_total	Counter	The total queuing time of read requests before they are sent. Unit: milliseconds.
node_volume_read_rtt_time_milliseconds_total	Counter	The total time consumption of read operations waiting for responses from servers. Unit: milliseconds.
node_volume_write_bytes_total	Counter	The total size of data that is successfully written to the NAS volume. Unit: bytes.
node_volume_write_recv_bytes_total	Counter	The total size of data that is received by write requests. Unit: bytes.
node_volume_write_completed_total	Counter	The total number of successful write operations.
node_volume_write_transmissions_total	Counter	The total number of write requests.
node_volume_write_timeouts_total	Counter	The total number of write timeouts.
node_volume_write_time_milliseconds_total	Counter	The total time consumption of write operations. Unit: milliseconds.
node_volume_write_queue_time_milliseconds_total	Counter	The total queuing time of write requests before they are sent. Unit: milliseconds.
node_volume_write_rtt_time_milliseconds_total	Counter	The total time consumption of write operations waiting for responses from servers. Unit: milliseconds.

Metrics related to OSS volumes

Important

The following metrics are reliant on csi-plugin. You must install CSI 1.22.14-820d8870-aliyun or later. For more information about the release notes for csi-plugin, see csi-plugin. When you update the CSI plug-in, you must update both csi-plugin and csi-provisioner.

Metric	Type	Description
node_volume_inode_bytes_total_counter	Counter	The total number of `inode`s in the OSS volume.
node_volume_inode_bytes_used_counter	Counter	The number of `inode`s used by the OSS volume.
node_volume_inode_bytes_available_counter	Counter	The number of remaining available `inode`s in the OSS volume.
node_volume_read_bytes_total_counter	Counter	The total size of data that is successfully read from the OSS volume. Unit: bytes.
node_volume_read_completed_total_counter	Counter	The total number of successful read operations.
node_volume_read_time_milliseconds_total_counter	Counter	The total time consumption of read operations. Unit: milliseconds.
node_volume_write_bytes_total_counter	Counter	The total size of data that is successfully written to the OSS volume. Unit: bytes.
node_volume_write_completed_total_counter	Counter	The total number of successful write operations.
node_volume_write_time_milliseconds_total_counter	Counter	The total time consumption of write operations. Unit: milliseconds.
node_volume_posix_mkdir_total_counter	Counter	Total number of POSIX `mkdir` operations.
node_volume_posix_rmdir_total_counter	Counter	Total number of POSIX `rmdir` operations.
node_volume_posix_opendir_total_counter	Counter	Total number of POSIX `opendir` operations.
node_volume_posix_readdir_total_counter	Counter	Total number of POSIX `readdir` operations.
node_volume_posix_read_total_counter	Counter	Total number of POSIX `read` operations.
node_volume_posix_write_total_counter	Counter	Total number of POSIX `write` operations.
node_volume_posix_flush_total_counter	Counter	Total number of POSIX `flush` operations.
node_volume_posix_fsync_total_counter	Counter	Total number of POSIX `fsync` operations.
node_volume_posix_release_total_counter	Counter	Total number of POSIX `release` operations.
node_volume_posix_create_total_counter	Counter	Total number of POSIX `create` operations.
node_volume_posix_open_total_counter	Counter	Total number of POSIX `open` operations.
node_volume_posix_access_total_counter	Counter	Total number of POSIX `access` operations.
node_volume_posix_rename_total_counter	Counter	Total number of POSIX `rename` operations.
node_volume_posix_chown_total_counter	Counter	Total number of POSIX `chown` operations.
node_volume_posix_chmod_total_counter	Counter	Total number of POSIX `chmod` operations.
node_volume_posix_truncate_total_counter	Counter	Total number of POSIX `truncate` operations.
node_volume_oss_put_object_total_counter	Counter	Total number of OSS `put` operations.
node_volume_oss_get_object_total_counter	Counter	Total number of OSS `get` operations.
node_volume_oss_head_object_total_counter	Counter	Total number of OSS `head` operations.
node_volume_oss_delete_object_total_counter	Counter	Total number of OSS `delete` operations.
node_volume_oss_post_object_total_counter	Counter	Total number of OSS `post` operations.
node_volume_hot_spot_read_file_top	Gauge	Hot files for `read` operations.
node_volume_hot_spot_write_file_top	Gauge	Hot files for `write` operations.
node_volume_hot_spot_head_file_top	Gauge	Hot files for `head` operations.
node_volume_backend_read_bytes_total_counter	Counter	The total size of data that is successfully read from the OSS backend. Unit: bytes.
node_volume_backend_write_bytes_total_counter	Counter	The total size of data that is successfully written to the OSS backend. Unit: bytes.
node_volume_backend_read_completed_total_counter	Counter	The total number of successful read operations on the OSS backend.
node_volume_backend_write_completed_total_counter	Counter	The total number of successful write operations on the OSS backend.
node_volume_backend_read_time_milliseconds_total_counter	Counter	The total time consumption of read operations on the OSS backend. Unit: milliseconds.
node_volume_backend_write_time_milliseconds_total_counter	Counter	The total time consumption of write operations on the OSS backend. Unit: milliseconds.
node_volume_backend_posix_getattr_total_counter	Counter	Total number of POSIX `getattr` operations on the OSS backend.
node_volume_backend_posix_getmode_total_counter	Counter	Total number of POSIX `getmode` operations on the OSS backend.
node_volume_backend_posix_access_total_counter	Counter	Total number of POSIX `access` operations on the OSS backend.
node_volume_backend_posix_lookup_total_counter	Counter	Total number of POSIX `lookup` operations on the OSS backend.
node_volume_backend_posix_mknod_total_counter	Counter	Total number of POSIX `mknod` operations on the OSS backend.
node_volume_backend_posix_remove_total_counter	Counter	Total number of POSIX `remove` operations on the OSS backend.
node_volume_backend_posix_setattr_total_counter	Counter	Total number of POSIX `setattr` operations on the OSS backend.
node_volume_backend_posix_link_total_counter	Counter	Total number of POSIX `link` operations on the OSS backend.
node_volume_backend_posix_readlink_total_counter	Counter	Total number of POSIX `readlink` operations on the OSS backend.
node_volume_backend_posix_statfs_total_counter	Counter	Total number of POSIX `statfs` operations on the OSS backend.
node_volume_backend_posix_rename_total_counter	Counter	Total number of POSIX `rename` operations on the OSS backend.
node_volume_backend_posix_readdir_total_counter	Counter	Total number of POSIX `readdir` operations on the OSS backend.

FUSE mount point metrics

Important

The following metrics are in canary release and require the csi-plugin component to be v1.32.2 or later. For component release notes, see csi-plugin. To upgrade, see Upgrade CSI components.

Metric	Type	Description
node_volume_mount_retry_count	Counter	The total number of retries for creating a FUSE mount point, triggered by timeouts or execution errors. A continuous increase in this value indicates a persistent issue with the mount operation, which can cause the application Pod to become stuck in the `ContainerCreating` state.
node_volume_mount_point_failover_count	Counter	The total number of successful failover operations that restore a FUSE mount point to a healthy state after a disconnection, such as a client process crash. This metric is supported only by specific client types. For unsupported clients, this value is always `0`.
node_volume_mount_point_status	Gauge	The real-time health status of the mount point. `0`: Healthy. The mount point is successfully mounted and the connection is stable. The application Pod can normally access data in the volume. `1`: Unhealthy. The mount operation failed, or the connection is lost or abnormal. This can cause the Pod to become stuck in the `ContainerCreating` state or experience I/O errors.
node_volume_last_fuse_client_exit_reason	Gauge	The timestamp and reason for the last unexpected exit of the FUSE client process. For example, `2025-11-06T07:19:32Z:: signal: killed` indicates that the process received a `kill` signal.

FAQ

Why is the storage monitoring dashboard empty after upgrading storage-operator to v1.33.1 or later?

After cnfs-controller became a managed add-on, the data collection path for monitoring metrics was changed. When upgrading storage-operator to v1.33.1 or later, make sure to also upgrade the related monitoring add-ons. Otherwise, the storage monitoring dashboard will fail to display metrics.

Upgrade ack-arms-prometheus to v1.1.37 or later. For instructions, see Manage add-ons.
Update the integration component for cluster monitoring to the latest version.