enhance the monitoring of block I/O throttling - Alibaba Cloud Linux

Alibaba Cloud Linux 2 (kernel version 4.19.81-17.al7 and later) and Alibaba Cloud Linux 3 add five interfaces to the blkio throttle subsystem. The upstream throttle policy tracks only bytes and operation counts — it has no latency or queue metrics. These interfaces fill that gap, letting you determine whether observed I/O latency comes from throttling or from underlying device contention.

Why these interfaces exist

Linux block I/O throttling limits bytes per second (BPS) or input/output operations per second (IOPS) for cgroups. The upstream throttle subsystem does not measure latency or queue depth, which means you cannot distinguish throttle-induced latency from device-level latency — especially when cgroup writeback is enabled.

The five Alibaba Cloud Linux interfaces add that observability directly under the throttle subsystem.

Available interfaces

All interfaces are at /sys/fs/cgroup/blkio/<cgroup>/, where <cgroup> is the target control group.

Interface	Description	Unit
`blkio.throttle.io_service_time`	Total time between request dispatch and request completion for I/O operations	nanoseconds
`blkio.throttle.io_wait_time`	Total time that I/O operations spend waiting in scheduler queues	nanoseconds
`blkio.throttle.io_completed`	Number of completed I/O operations; use with the two time interfaces to calculate average latency	—
`blkio.throttle.total_io_queued`	Number of I/O operations that were throttled; compare periodic readings to determine whether I/O latency is related to throttling	—
`blkio.throttle.total_bytes_queued`	Number of I/O bytes that were throttled	bytes

Calculate average write latency

This example monitors write I/O latency for the vdd disk (device 254:48) in the blkcg1 cgroup over a 5-second interval.

Prerequisites

Before you begin, ensure that you have:

A cgroup (blkcg1) with a BPS or IOPS throttle limit configured
The major:minor number of the disk to monitor (run lsblk -o NAME,MAJ:MIN to find it)
Root access to the system

Run the latency measurement script

Run the following script as root:

#!/bin/bash

CGROUP_PATH="/sys/fs/cgroup/blkio/blkcg1"
DEVICE="254:48"
INTERVAL=5

# Collect T1 readings
write_wait_time1=$(cat "$CGROUP_PATH/blkio.throttle.io_wait_time" | grep -w "$DEVICE Write" | awk '{print $3}')
write_service_time1=$(cat "$CGROUP_PATH/blkio.throttle.io_service_time" | grep -w "$DEVICE Write" | awk '{print $3}')
write_completed1=$(cat "$CGROUP_PATH/blkio.throttle.io_completed" | grep -w "$DEVICE Write" | awk '{print $3}')

sleep "$INTERVAL"

# Collect T2 readings (T1 + 5 seconds)
write_wait_time2=$(cat "$CGROUP_PATH/blkio.throttle.io_wait_time" | grep -w "$DEVICE Write" | awk '{print $3}')
write_service_time2=$(cat "$CGROUP_PATH/blkio.throttle.io_service_time" | grep -w "$DEVICE Write" | awk '{print $3}')
write_completed2=$(cat "$CGROUP_PATH/blkio.throttle.io_completed" | grep -w "$DEVICE Write" | awk '{print $3}')

# Calculate average I/O latency over the interval
avg_delay=$(echo "((write_wait_time2 + write_service_time2) - (write_wait_time1 + write_service_time1)) / (write_completed2 - write_completed1)" | bc)

echo "Average write latency over ${INTERVAL}s: ${avg_delay} ns"

The script applies the following formula:

avg_delay = (total_io_duration_T2 - total_io_duration_T1) / (completed_ops_T2 - completed_ops_T1)

where total_io_duration = io_wait_time + io_service_time.

Variable	Interface	Description
`write_wait_time<N>`	`blkio.throttle.io_wait_time`	Duration of throttling at the block I/O throttling layer
`write_service_time<N>`	`blkio.throttle.io_service_time`	Cumulative time between request dispatch and completion for write I/O, in nanoseconds
`write_completed<N>`	`blkio.throttle.io_completed`	Cumulative number of completed write I/O operations

Run the script periodically — for example, with watch or a cron job — to track latency trends over time.