blk-iocost weight-based throttling - Alibaba Cloud Linux

Blk-iocost weight-based throttling enhances cgroup I/O subsystem (blkcg) disk throttling in Alibaba Cloud Linux. It allocates block device I/O bandwidth by application or process priority. Configure weight values to limit per-application or per-process I/O bandwidth.

Note

The Alibaba Cloud Linux kernel supports both cgroup v1 and v2 interfaces for blk-iocost, but only one is active at a time. Run stat -fc %T /sys/fs/cgroup to check which version is active.

A return value of tmpfs indicates the cgroup v1 interface.
A return value of cgroup2fs indicates the cgroup v2 interface.

Supported operating systems

Alibaba Cloud Linux 2 with kernel version 4.19.81-17 or later
Alibaba Cloud Linux 3

Use the cost.qos interface

The cost.qos interface enables or disables blk-iocost and rate-limits I/O quality of service (QoS) based on latency and weight. It exists only in the root blkcg group, and its name varies between cgroup versions.

cgroup v1: blkio.cost.qos
cgroup v2: io.cost.qos

Configuration

Each line starts with the device number in MAJ:MIN format (run lsblk | grep <Cloud_Disk_Name> to find it), followed by these parameters:

Parameter

Description

enable

Enables or disables blk-iocost.

0: Default. Disables the blk-iocost feature.
1: Enables the blk-iocost feature.

ctrl

The control mode.

auto: Automatically detects the device type and uses built-in parameters.
Important
When ctrl is set to auto on an ECS instance with a solid-state device (standard SSD, ESSD, or NVMe SSD local disk), set the rotational property of the disk to 0 so that blk-iocost can accurately estimate I/O cost. Example:
```
sudo sh -c 'echo 0 > /sys/block/<DISK_NAME>/queue/rotational'
```
Replace <DISK_NAME> with the actual disk name.
user: You must manually set the following control parameters:
- rpct: the read latency percentile. Valid values: 0 to 100.
- rlat: the read latency threshold. Unit: microseconds.
- wpct: the write latency percentile. Valid values: 0 to 100.
- wlat: the write latency threshold. Unit: microseconds.
- min: the minimum scaling percentage. Valid values: 1 to 10000.
- max: the maximum scaling percentage. Valid values: 1 to 10000.

Enable the blk-iocost feature

This example enables blk-iocost for device 254:48 in user mode. The disk is saturated when 95% of read/write latencies (rlat and wlat) exceed 5000 μs (5 ms), and the kernel adjusts the dispatch rate between 50% and 150%.

cgroup v1 interface

sudo sh -c 'echo "254:48 enable=1 ctrl=user rpct=95.00 rlat=5000 wpct=95.00 wlat=5000 min=50.00 max=150.00" > /sys/fs/cgroup/blkio/blkio.cost.qos'

cgroup v2 interface

sudo sh -c 'echo "254:48 enable=1 ctrl=user rpct=95.00 rlat=5000 wpct=95.00 wlat=5000 min=50.00 max=150.00" > /sys/fs/cgroup/io.cost.qos'

Use the cost.model interface

The cost.model interface configures the I/O cost model. It exists only in the root blkcg group, and its name varies between cgroup versions.

cgroup v1: blkio.cost.model
cgroup v2: io.cost.model

Configuration

Each line starts with the device number in MAJ:MIN format (run lsblk | grep <Cloud_Disk_Name> to find it), followed by these parameters:

Parameter

Description

ctrl

The control mode.

auto: Automatically optimizes I/O scheduling based on workload.
Important
When ctrl is set to auto on an ECS instance with a solid-state device (standard SSD, ESSD, or NVMe SSD local disk), set the rotational property of the disk to 0 so that blk-iocost can accurately estimate I/O cost. Example:
```
sudo sh -c 'echo 0 > /sys/block/<DISK_NAME>/queue/rotational'
```
Replace <DISK_NAME> with the actual disk name.
user: Manually specify model parameters.

model

Currently only the linear model is supported. When model is set to linear, define the following parameters:

[r|w]bps: the maximum sequential I/O throughput. Unit: bytes/second.
[r|w]seqiops: the sequential input/output operations per second (IOPS) limit.
[r|w]randiops: the random IOPS limit.

Note
Use the tools/cgroup/iocost_coef_gen.py script in the kernel source to generate these parameters and write them to the cost.model interface.

Set a cost model

This example sets a linear cost model for device 254:48 with user-defined parameters.

cgroup v1 interface

sudo sh -c 'echo "254:48 ctrl=user model=linear rbps=2706339840 rseqiops=89698 rrandiops=110036 wbps=1063126016 wseqiops=135560 wrandiops=130734" > /sys/fs/cgroup/blkio/blkio.cost.model'

cgroup v2 interface

sudo sh -c 'echo "254:48 ctrl=user model=linear rbps=2706339840 rseqiops=89698 rrandiops=110036 wbps=1063126016 wseqiops=135560 wrandiops=130734" > /sys/fs/cgroup/io.cost.model'

Use the weight/cost.weight interface

This interface controls I/O resource allocation through a weight value between 1 and 10000. The interface name varies by cgroup version and OS. These files exist only in blkcg subgroups.

Alibaba Cloud Linux 3
- cgroup v1: blkio.cost.weight
- cgroup v2: io.weight
Alibaba Cloud Linux 2
- cgroup v1: blkio.cost.weight
- cgroup v2: io.cost.weight

Configuration

To modify the default weight of a blkcg, set a weight value for the interface: <weight>.
To modify the weight of a blkcg on a specific device, use the following format: MAJ:MIN <weight>.

Modify weights

After you enable blk-iocost, these examples create control groups (blkcg1 for cgroup v1, cg1 for cgroup v2) and set both the default weight and the device-specific weight for 254:48 to 50.

cgroup v1 interface

sudo mkdir /sys/fs/cgroup/blkio/blkcg1    # Create the control group blkcg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/blkio/blkcg1/blkio.cost.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/blkio/blkcg1/blkio.cost.weight'    # Set the weight on the disk to 50

cgroup v2 interface

Alibaba Cloud Linux 3

sudo mkdir /sys/fs/cgroup/cg1    # Create the control group cg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/cg1/io.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/cg1/io.weight'    # Set the weight on the disk to 50

Alibaba Cloud Linux 2

sudo mkdir /sys/fs/cgroup/cg1    # Create the control group cg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/cg1/io.cost.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/cg1/io.cost.weight'    # Set the weight on the disk to 50

Common monitoring tools

The following tools help monitor and tune I/O performance under blk-iocost.

iocost monitor script

The tools/cgroup/iocost_monitor.py script uses the drgn debugger to access kernel parameters and output I/O performance data. Steps:

Install the drgn debugger.
```
sudo pip3 install drgn
```

Download the iocost_monitor.py script.

wget https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/cgroup/iocost_monitor.py

Run the iocost_monitor.py script.

In this example, the vdd disk is used.

sudo python3 ./iocost_monitor.p vdd

Example output:

vdd RUN  per=500.0ms cur_per=3930.839:v14620.321 busy= +1 vrate=6136.22% params=hdd
                          active    weight      hweight% inflt% dbt  delay usages%
blkcg1                       *    50/   50   9.09/  9.09   0.00   0  0*000 009:009:009
blkcg2                       *   500/  500  90.91/ 90.91   0.00   0  0*000 089:091:092

blkio.cost.stat interface (cgroup v1)

The blkio.cost.stat interface (cgroup v1) records QoS data for each controlled device:

cat /sys/fs/cgroup/blkio/blkcg1/blkio.cost.stat

Example output:

254:48 is_active=1 active=50 inuse=50 hweight_active=5957 hweight_inuse=5957 vrate=159571

ftrace tool

Use ftrace to capture blk-iocost scheduler decisions and trace I/O request handling for detailed performance analysis. Steps:

Set the enable attribute to 1 to enable the ftrace tool.

sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/events/iocost/enable'

View the output information.

sudo cat /sys/kernel/debug/tracing/trace_pipe

Example output:

    dd-1593  [008] d...   688.565349: iocost_iocg_activate: [vdd:/blkcg1] now=689065289:57986587662878 vrate=137438 period=22->22 vtime=0->57986365150756 weight=50/50 hweight=65536/65536
    dd-1593  [008] d.s.   688.575374: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.608369: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
    dd-1594  [006] d...   688.620002: iocost_iocg_activate: [vdd:/blkcg2] now=689119946:57994099611644 vrate=137438 period=22->26 vtime=0->57993412421644 weight=250/250 hweight=65536/65536
<idle>-0     [008] d.s.   688.631367: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.642368: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.653366: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.664366: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1