Fix NFS read_ahead_kb performance Alibaba Cloud Linux 3 - Alibaba Cloud Linux

On Alibaba Cloud Linux 3, NFS file reads via system calls such as read and copy_file_range can be significantly slower than on Alibaba Cloud Linux 2 due to a reduced default read-ahead window size in kernel 5.10.

Problem description

Symptom

NFS file reads on Alibaba Cloud Linux 3 may show degraded performance:

System calls such as read and copy_file_range take unusually long to read large NFS files.
The dd command reads slower from an NFS mount point on Alibaba Cloud Linux 3 than on Alibaba Cloud Linux 2:
```
dd if=<nfs_mntpoint>/<testfile> of=/dev/null bs=1M
```
Note
This reads testfile from the NFS mount point to /dev/null. The dd output shows bytes read and time elapsed, from which you can calculate the read rate.

Impact

Affected ECS instances:

Image: aliyun_3_x64_20G_alibase_20210415.vhd and later.
Kernel: 5.10.23-4.al8.x86_64 and later.
File system: Reading files from a mounted NFS file system.

Cause

The read_ahead_kb kernel parameter defines the read-ahead window size for a block device. Read-ahead preloads predicted data into memory so subsequent reads avoid disk I/O, reducing latency.

Before kernel 5.4, NFS read-ahead was based on the rsize mount parameter (rsize = size of each NFS read request). The default read_ahead_kb was 15 × rsize. Alibaba Cloud Linux 2 (kernel 4.19) defaults to rsize = 1,024 KB, giving a read_ahead_kb of 15,360 KB.
A kernel 5.4 commit changed this: read_ahead_kb is now based on VM_READAHEAD_PAGES instead of rsize. Alibaba Cloud Linux 3 (kernel 5.10) defaults to read_ahead_kb = 128 KB.

This significantly smaller read-ahead window causes lower read performance on Alibaba Cloud Linux 3. Adjust the value to restore performance.

Important

A larger read-ahead window improves sequential read performance but may waste memory on random reads. Evaluate your workload to determine the optimal read_ahead_kb value.

Solution

Modify read_ahead_kb using one of the following methods.

Use echo (single file system)

Check the current read-ahead value for the NFS device.
```
cat /sys/class/bdi/$(mountpoint -d <nfs_mountpoint>)/read_ahead_kb
```
Replace <nfs_mountpoint> with your NFS mount point path. Run cat /proc/self/mountinfo to find it.
Set the read-ahead value for the NFS backing device.
```
sudo sh -c 'echo <num> > /sys/class/bdi/<major>:<minor>/read_ahead_kb'
```
Replace:
- <num>: Read-ahead window size in KB.
- <major>:<minor>: Device numbers of the NFS file system. Run sudo mountpoint -d <nfs_mountpoint> to obtain them.
For example:
```
sudo sh -c 'echo 15360 > /sys/class/bdi/0:422/read_ahead_kb'
```
Note
For multiple NFS mounts, run this command for each device.

Use udev rules (multiple file systems)

Add a udev rule to automatically set the read-ahead parameter for all current and future NFS mounts.

Note

udev is a Linux subsystem that automates device management by listening for kernel events and triggering actions when devices are added.

Create or edit the udev rules file in /etc/udev/rules.d/:
```
sudo vim /etc/udev/rules.d/99-nfs.rules
```

Add the following udev rule:

This sets read_ahead_kb to 15,360 KB. Adjust as needed.

SUBSYSTEM=="bdi", ACTION=="add", PROGRAM="/bin/awk -v bdi=$kernel 'BEGIN{ret=1} {if ($4 == bdi) {ret=0}} END{exit ret}' /proc/fs/nfsfs/volumes", ATTR{read_ahead_kb}="15360"

Save and close the file.
Reload the udev rules.
```
sudo udevadm control --reload
```
Trigger the rule for existing devices.
```
sudo udevadm trigger -c add -s bdi
```

Edit nfs.conf (nfs-utils 2.3.3-57.0.1.al8.1 or later)

Requires nfs-utils 2.3.3-57.0.1.al8.1 or later. Set read_ahead_kb in /etc/nfs.conf. Check your nfs-utils version: rpm -qa | grep nfs-utils.

Open and edit the NFS configuration file.
```
sudo vim /etc/nfs.conf 
```
Set the read-ahead values and save the file.
```
[nfsrahead]
nfs=15000
nfs4=16000
```
The nfs parameter applies to NFSv3, and nfs4 to NFSv4. Check your version: mount -v | grep nfs.

Unmount and remount existing NFS mounts to apply the changes.

sudo umount <nfs_mountpoint>
sudo mount -t nfs -o vers=<NFS protocol version> <NFS server address> <nfs_mountpoint>