What do I do if the buffer I/O write performance of an Ext4 file system on an Alibaba Cloud Linux 2 ECS instance does not meet expectations? - Alibaba Cloud Linux

If your Alibaba Cloud Linux 2 Elastic Compute Service (ECS) instance runs an Ext4 file system mounted with the dioread_nolock and nodelalloc options, you may see buffer I/O write throughput fall well below the rated block storage performance. This article explains the affected configurations, root cause, and how to fix the issue.

Check whether your instance is affected

Your instance is affected only if all three of the following conditions are true:

Image version: from aliyun-2.1903-x64-20G-alibase-20190327.vhd (inclusive) to aliyun_2_1903_x64_20G_alibase_20220525.vhd (exclusive).
Kernel version: from kernel-4.19.24-9.al7 (inclusive) to kernel-4.19.91-26.al7.x86_64 (exclusive). Run uname -r to check.
Mount options: the Ext4 file system is mounted with both dioread_nolock and nodelalloc.

dioread_nolock and nodelalloc are non-default mount options. If you did not explicitly set these options, your instance is likely unaffected.

To confirm the mount options, run the following commands:

Identify the disk partition that contains the target write directory. Replace <$DIR> with your target directory path.
```
df <$DIR> | grep -v Filesystem | awk '{ print $1 }'
```
Check the file system type and mount options of that partition. Replace <$Partition> with the partition name from the previous step.
```
mount | grep -w <$Partition> | grep ext4 | grep -w dioread_nolock | grep -w nodelalloc
```
If the command returns output, your instance is affected.

Symptoms

When the issue occurs, buffer I/O write throughput drops significantly — typically to around 30 MB/s — well below the expected block storage performance of the ECS instance.

The following iostat -xm 1 output is a typical example of the degraded state (see the wMB/s column):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00   12.77    0.00    0.00   87.23

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
vdb               0.00  7194.00    0.00   57.00     0.00    28.05  1008.00     0.02   17.81    0.00   17.81   0.39   2.20

The issue is reproducible with:

Large file copies: cp <$LargeFiles> /mnt/badfile (use files larger than 2 GiB to trigger the issue)
`dd` without synchronization flags: dd if=/dev/zero of=/mnt/badfile bs=10M count=1000

Root cause

When an Ext4 file system is mounted with both dioread_nolock and nodelalloc, the kernel generates a large number of 4-KB dirty pages called *unwritten extents*. A defect in the Ext4 file system processing logic prevents these pages from being merged into huge pages before writeback. The ext4_writepages function then processes each page individually, spending excessive time searching and mapping 4-KB dirty pages — resulting in extremely low write throughput.

To observe this behavior, use the perf tool to monitor the kernel's page cache writeback process.

Solutions

Two solutions are available. Solution 1 takes effect immediately and does not require a reboot.

Solution 1: Remount the Ext4 file system without the problematic options

This solution takes effect immediately without restarting the instance.

Remount the Ext4 file system using the delalloc option (the default), dropping dioread_nolock and nodelalloc. Replace the placeholders with your actual values.

Placeholder	Description	How to find
`<$Device>`	Device name of the Ext4 file system	Check the NAME column in `lsblk` output
`<$MountPoint>`	Mount point of the Ext4 file system	Use an existing empty directory, or create one with `sudo mkdir -p <new-directory>`

sudo mount -o remount,delalloc <$Device> <$MountPoint>

Persist the change across reboots by removing nodelalloc from the /etc/fstab entry for this file system. Without this step, the mount options revert to nodelalloc after a reboot. The default delalloc option does not need to be explicitly specified in /etc/fstab — Ext4 file systems use it by default.
Verify that write performance has recovered. Run iostat -xm 1 and confirm the wMB/s value matches the expected block storage performance of your ECS instance.

Solution 2: Upgrade the kernel

This solution fixes the underlying defect permanently but requires a restart.

Warning

Kernel upgrades may cause compatibility and stability issues. Review the release notes for Alibaba Cloud Linux 2 before upgrading. Back up critical instance data and schedule the restart during off-peak hours, as the restart temporarily stops the instance and may interrupt running services.

Upgrade to the latest kernel version.
```
sudo yum update kernel
```
Restart the instance for the new kernel to take effect.
```
sudo reboot
```
After the instance restarts, verify that write performance has recovered. Run iostat -xm 1 and confirm the wMB/s value matches the expected block storage performance of your ECS instance.