Container storage monitoring - Container Service for Kubernetes

After you enable Managed Service for Prometheus, you can monitor internal storage (root file systems and ephemeral storage) and external storage (cloud disks, NAS, and OSS volumes). Container storage monitoring exposes multi-dimensional storage metrics that are free to use.

Monitor internal cluster storage

Internal storage includes the root file system and ephemeral storage.

Root file system: stores container runtime data and configurations. Managed Service for Prometheus (Prometheus) monitors its usage, utilization, and distribution.
Ephemeral storage: stores temporary files and cache from the container runtime. Prometheus monitors its usage, utilization, and distribution.

Metrics related to the root file system

Important

For clusters running Kubernetes 1.22 or later, update the Container Storage Interface (CSI) plug-in to 1.28.3-eb95171-aliyun or later. See the csi-plugin release notes. Always update both csi-plugin and csi-provisioner together.

Metric	Type	Description
container_fs_limit_bytes	Gauge	Total storage capacity of the root file system. Unit: bytes.
container_fs_usage_bytes	Gauge	Storage usage of the root file system. Unit: bytes.
container_fs_available_bytes	Gauge	Available storage capacity of the root file system. Unit: bytes.
container_fs_inodes_total	Gauge	Total inodes in the root file system.
container_fs_inodes_used	Gauge	Used inodes in the root file system.
container_fs_inodes_free	Gauge	Available inodes in the root file system.
container_fs_reads_bytes_total	Gauge	Total data read from the root file system. Unit: bytes.
container_fs_writes_bytes_total	Gauge	Total data written to the root file system. Unit: bytes.

Metrics related to ephemeral storage

Important

These metrics require csi-plugin 1.28.3-eb95171-aliyun or later. See the csi-plugin release notes. Always update both csi-plugin and csi-provisioner together.

Metric	Type	Description
ephemeral_storage_pod_limit_bytes	Gauge	Total ephemeral storage capacity in the pod. Unit: bytes.
ephemeral_storage_pod_usage_bytes	Gauge	Ephemeral storage usage in the pod. Unit: bytes.
ephemeral_storage_pod_available_bytes	Gauge	Available ephemeral storage in the pod. Unit: bytes.
ephemeral_storage_pod_inodes_total	Gauge	Total inodes in the pod's ephemeral storage.
ephemeral_storage_pod_inodes_used	Gauge	Used inodes in the pod's ephemeral storage.
ephemeral_storage_pod_inodes_free	Gauge	Available inodes in the pod's ephemeral storage.

Monitor external cluster storage

Metrics related to disk volumes

Important

These metrics require csi-plugin 1.18.8.46-afb19e46-aliyun or later. See the csi-plugin release notes. Always update both csi-plugin and csi-provisioner together.

Metric	Type	Description
node_volume_capacity_bytes_total	Counter	Total disk capacity. Unit: bytes.
node_volume_capacity_bytes_used	Counter	Disk capacity usage. Unit: bytes.
node_volume_capacity_bytes_available	Counter	Available disk capacity. Unit: bytes.
node_volume_inodes_total	Counter	Total inodes on the disk.
node_volume_inodes_used	Counter	Used inodes on the disk.
node_volume_inodes_available	Counter	Available inodes on the disk.
node_volume_read_bytes_total	Counter	Total data successfully read from the disk. Unit: bytes.
node_volume_read_completed_total	Counter	Total successful read operations.
node_volume_read_merged_total	Counter	Total kernel read merge operations.
node_volume_read_time_milliseconds_total	Counter	Total read operation time. Unit: milliseconds.
node_volume_write_bytes_total	Counter	Total data successfully written to the disk. Unit: bytes.
node_volume_write_completed_total	Counter	Total successful write operations.
node_volume_write_merged_total	Counter	Total kernel write merge operations.
node_volume_write_time_milliseconds_total	Counter	Total write operation time. Unit: milliseconds.
node_volume_io_now	Gauge	In-progress I/O operations.
node_volume_io_time_seconds_total	Counter	Total I/O operation time. Unit: seconds.

Metrics related to NAS volumes

Important

These metrics require csi-plugin 1.18.8.46-afb19e46-aliyun or later. See the csi-plugin release notes. Always update both csi-plugin and csi-provisioner together.

Metric	Type	Description
node_volume_capacity_bytes_total	Counter	Total NAS volume capacity. Unit: bytes.
node_volume_capacity_bytes_used	Counter	NAS volume capacity usage. Unit: bytes.
node_volume_capacity_bytes_available	Counter	Available NAS volume capacity. Unit: bytes.
node_volume_read_bytes_total	Counter	Total data successfully read from the NAS volume. Unit: bytes.
node_volume_read_sent_bytes_total	Counter	Total data sent by read requests. Unit: bytes.
node_volume_read_completed_total	Counter	Total successful read operations.
node_volume_read_transmissions_total	Counter	Total read requests.
node_volume_read_timeouts_total	Counter	Total request timeouts.
node_volume_read_time_milliseconds_total	Counter	Total read operation time. Unit: milliseconds.
node_volume_read_queue_time_milliseconds_total	Counter	Total queue time for read requests. Unit: milliseconds.
node_volume_read_rtt_time_milliseconds_total	Counter	Total server response wait time for reads. Unit: milliseconds.
node_volume_write_bytes_total	Counter	Total data successfully written to the NAS volume. Unit: bytes.
node_volume_write_recv_bytes_total	Counter	Total data received by write requests. Unit: bytes.
node_volume_write_completed_total	Counter	Total successful write operations.
node_volume_write_transmissions_total	Counter	Total write requests.
node_volume_write_timeouts_total	Counter	Total write timeouts.
node_volume_write_time_milliseconds_total	Counter	Total write operation time. Unit: milliseconds.
node_volume_write_queue_time_milliseconds_total	Counter	Total queue time for write requests. Unit: milliseconds.
node_volume_write_rtt_time_milliseconds_total	Counter	Total server response wait time for writes. Unit: milliseconds.

Metrics related to OSS volumes

Important

These metrics require csi-plugin 1.22.14-820d8870-aliyun or later. See the csi-plugin release notes. Always update both csi-plugin and csi-provisioner together.

Metric	Type	Description
node_volume_inode_bytes_total_counter	Counter	Total `inode`s in the OSS volume.
node_volume_inode_bytes_used_counter	Counter	`inode`s used by the OSS volume.
node_volume_inode_bytes_available_counter	Counter	Available `inode`s in the OSS volume.
node_volume_read_bytes_total_counter	Counter	Total data successfully read from the OSS volume. Unit: bytes.
node_volume_read_completed_total_counter	Counter	Total successful read operations.
node_volume_read_time_milliseconds_total_counter	Counter	Total read operation time. Unit: milliseconds.
node_volume_write_bytes_total_counter	Counter	Total data successfully written to the OSS volume. Unit: bytes.
node_volume_write_completed_total_counter	Counter	Total successful write operations.
node_volume_write_time_milliseconds_total_counter	Counter	Total write operation time. Unit: milliseconds.
node_volume_posix_mkdir_total_counter	Counter	Total number of POSIX `mkdir` operations.
node_volume_posix_rmdir_total_counter	Counter	Total number of POSIX `rmdir` operations.
node_volume_posix_opendir_total_counter	Counter	Total number of POSIX `opendir` operations.
node_volume_posix_readdir_total_counter	Counter	Total number of POSIX `readdir` operations.
node_volume_posix_read_total_counter	Counter	Total number of POSIX `read` operations.
node_volume_posix_write_total_counter	Counter	Total number of POSIX `write` operations.
node_volume_posix_flush_total_counter	Counter	Total number of POSIX `flush` operations.
node_volume_posix_fsync_total_counter	Counter	Total number of POSIX `fsync` operations.
node_volume_posix_release_total_counter	Counter	Total number of POSIX `release` operations.
node_volume_posix_create_total_counter	Counter	Total number of POSIX `create` operations.
node_volume_posix_open_total_counter	Counter	Total number of POSIX `open` operations.
node_volume_posix_access_total_counter	Counter	Total number of POSIX `access` operations.
node_volume_posix_rename_total_counter	Counter	Total number of POSIX `rename` operations.
node_volume_posix_chown_total_counter	Counter	Total number of POSIX `chown` operations.
node_volume_posix_chmod_total_counter	Counter	Total number of POSIX `chmod` operations.
node_volume_posix_truncate_total_counter	Counter	Total number of POSIX `truncate` operations.
node_volume_oss_put_object_total_counter	Counter	Total number of OSS `put` operations.
node_volume_oss_get_object_total_counter	Counter	Total number of OSS `get` operations.
node_volume_oss_head_object_total_counter	Counter	Total number of OSS `head` operations.
node_volume_oss_delete_object_total_counter	Counter	Total number of OSS `delete` operations.
node_volume_oss_post_object_total_counter	Counter	Total number of OSS `post` operations.
node_volume_hot_spot_read_file_top	Gauge	Hot files for `read` operations.
node_volume_hot_spot_write_file_top	Gauge	Hot files for `write` operations.
node_volume_hot_spot_head_file_top	Gauge	Hot files for `head` operations.
node_volume_backend_read_bytes_total_counter	Counter	Total data successfully read from the OSS backend. Unit: bytes.
node_volume_backend_write_bytes_total_counter	Counter	Total data successfully written to the OSS backend. Unit: bytes.
node_volume_backend_read_completed_total_counter	Counter	Total successful read operations on the OSS backend.
node_volume_backend_write_completed_total_counter	Counter	Total successful write operations on the OSS backend.
node_volume_backend_read_time_milliseconds_total_counter	Counter	Total read operation time on the OSS backend. Unit: milliseconds.
node_volume_backend_write_time_milliseconds_total_counter	Counter	Total write operation time on the OSS backend. Unit: milliseconds.
node_volume_backend_posix_getattr_total_counter	Counter	Total number of POSIX `getattr` operations on the OSS backend.
node_volume_backend_posix_getmode_total_counter	Counter	Total number of POSIX `getmode` operations on the OSS backend.
node_volume_backend_posix_access_total_counter	Counter	Total number of POSIX `access` operations on the OSS backend.
node_volume_backend_posix_lookup_total_counter	Counter	Total number of POSIX `lookup` operations on the OSS backend.
node_volume_backend_posix_mknod_total_counter	Counter	Total number of POSIX `mknod` operations on the OSS backend.
node_volume_backend_posix_remove_total_counter	Counter	Total number of POSIX `remove` operations on the OSS backend.
node_volume_backend_posix_setattr_total_counter	Counter	Total number of POSIX `setattr` operations on the OSS backend.
node_volume_backend_posix_link_total_counter	Counter	Total number of POSIX `link` operations on the OSS backend.
node_volume_backend_posix_readlink_total_counter	Counter	Total number of POSIX `readlink` operations on the OSS backend.
node_volume_backend_posix_statfs_total_counter	Counter	Total number of POSIX `statfs` operations on the OSS backend.
node_volume_backend_posix_rename_total_counter	Counter	Total number of POSIX `rename` operations on the OSS backend.
node_volume_backend_posix_readdir_total_counter	Counter	Total number of POSIX `readdir` operations on the OSS backend.

FUSE mount point metrics

Important

These metrics are in canary release and require csi-plugin v1.32.2 or later. See the csi-plugin release notes. To upgrade, see Upgrade CSI components.

Metric	Type	Description
node_volume_mount_retry_count	Counter	Total retries for FUSE mount point creation due to timeouts or execution errors. A sustained increase indicates a persistent mount issue that can cause the application Pod to stay in `ContainerCreating` state.
node_volume_mount_point_failover_count	Counter	Total successful failover operations that restore a FUSE mount point after disconnection, such as a client process crash. This metric is supported only by specific client types. For unsupported clients, this value is always `0`.
node_volume_mount_point_status	Gauge	Real-time mount point health status. `0`: Healthy. Mount point is active and connection is stable. The Pod can access volume data normally. `1`: Unhealthy. Mount failed or connection is lost/abnormal. Can cause the Pod to stay in `ContainerCreating` state or experience I/O errors.
node_volume_last_fuse_client_exit_reason	Gauge	Timestamp and reason for the last unexpected FUSE client exit. For example, `2025-11-06T07:19:32Z:: signal: killed` means the process received a `kill` signal.

FAQ

Why is the storage monitoring dashboard empty after upgrading storage-operator to v1.33.1 or later?

After cnfs-controller became a managed add-on, the data collection path for monitoring metrics was changed. When upgrading storage-operator to v1.33.1 or later, make sure to also upgrade the related monitoring add-ons. Otherwise, the storage monitoring dashboard will fail to display metrics.

Upgrade ack-arms-prometheus to v1.1.37 or later. For instructions, see Manage add-ons.
Update the integration component for cluster monitoring to the latest version.