Test the IOPS performance of an ESSD - Elastic Compute Service

Use fio to benchmark IOPS on a raw ESSD, which supports up to 1,000,000 random read/write IOPS with 25 GE and RDMA.

Prerequisites

Test tool: fio (flexible I/O tester).

Note
fio is an open source I/O benchmarking tool that can test block storage performance metrics such as random and sequential read/write operations.
Instance type: ecs.g7se.32xlarge (recommended). See General-purpose instance families (g series).
Image: A recent Alibaba Cloud Linux public image. Alibaba Cloud Linux 3 is used in the following examples.

Note
ESSDs may not achieve expected IOPS on certain Linux distributions. Attach ESSDs to ECS instances that use Alibaba Cloud Linux 3 images.
ESSD:
- Test raw disks with fio to obtain actual disk performance.
- Use a PL3 ESSD. See ESSDs.
Important
- If a block storage device contains partitions, file systems, or data, fio may corrupt the file systems and cause data loss. Create snapshots before testing. See Create snapshot manually.
- Do not test with the system disk or any data disk that contains data. Use a newly created, uninitialized, empty data disk instead.
- Test results are obtained in a test environment and are for reference only. In production, cloud disk performance may vary due to factors such as network conditions and concurrent access.
- After testing the new disk:
  - This item is reserved and cannot be used directly. See Re-initialize a data disk.
  - If you do not need the disk, detach and then release it.

Procedure

Connect to an ECS instance.

For more information, see Log on to a Linux instance using Workbench.
Query block storage device names:
```
sudo fdisk -lu
```
The output shows three block storage devices: system disk /dev/vda and two data disks /dev/vdb and /dev/vdc.
Check whether the block storage devices have partitions and file systems:
```
sudo blkid
```
The output shows that /dev/vda and /dev/vdb have partitions and file systems. /dev/vdc is absent from the output, indicating it has no partitions or file systems.
Back up data on the block storage devices before testing to prevent data loss. See Create snapshot manually.

Note
Snapshots incur charges. See Snapshot billing.
Install the libaio library and fio based on your operating system.
Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, or CentOS 6 or later

Note
CentOS 6 reached end of life (EOL). In accordance with Linux community rules, all content was removed from the following CentOS 6 repository address: http://mirror.centos.org/centos-6/. If you continue to use the default CentOS 6 repository on Alibaba Cloud, an error is reported. To use specific installation packages of CentOS 6, change the CentOS 6 repository address. For more information, see How do I change CentOS 6 repository addresses?
```
sudo yum install libaio libaio-devel fio -y
```
Debian 9 or later, or Ubuntu 14 or later

Important
Debian 9 and Debian 10 reached their EOL. If your instance runs Debian 9 or Debian 10, change the repository addresses.
```
sudo apt-get update
sudo apt-get install libaio* fio -y
```
Change to the /tmp directory:
```
cd /tmp
```
Create the test100w.sh script:
```
sudo vim test100w.sh
```

Paste the following content to the test100w.sh script.

See the Details of the test100w.sh script section.

#!/bin/bash

DEV_NODE=your_device
DEV_NAME=/dev/$DEV_NODE

function CheckHasFS
{
    local device=$1 # The device path.

    # Check whether the device exists.
    if [ !  -b "$device" ]; then
        echo "Error: The $device device does not exist."
        exit 1
    fi

    # Run the blkid command to check the partition table and file system type.
    local pt_type=$(sudo blkid -o value -s PTTYPE "$device")
    local fs_type=$(sudo blkid -o value -s TYPE "$device")

    if [ -n "$pt_type" ] || [ -n "$fs_type" ]; then
        return 1
    else
        return 0
    fi
}

CheckHasFS "$DEV_NAME"
if [ $?  -eq 1 ]; then
    echo "The $DEV_NAME device contains a partition table or a file system. The fio script is stopped."
    exit 1
fi

function RunFio
{
 numjobs=$1   # The number of test threads. In this example, the value is 10.
 iodepth=$2   # The maximum number of concurrent I/O requests. In this example, the value is 64.
 bs=$3        # The data block size per I/O. In this example, the value is 4k.
 rw=$4        # The read and write policy. In this example, the value is randwrite.
 size=$5
 filename=$6  # The name of the test file. In this example, the value is /dev/your_device.
 nr_cpus=`cat /proc/cpuinfo |grep "processor" |wc -l`
 if [ $nr_cpus -lt $numjobs ];then
     echo "The value of the numjobs parameter is greater than the number of CPU cores. The test is stopped."
     exit -1
 fi
 let nu=$numjobs+1
 cpulist=""
 for ((i=1;i<10;i++))
 do
     list=`cat /sys/block/$DEV_NODE/mq/*/cpu_list | awk '{if(i<=NF) print $i;}' i="$i" | tr -d ',' | tr '\n' ','`
     if [ -z $list ];then
         break
     fi
     cpulist=${cpulist}${list}
 done
 spincpu=`echo $cpulist | cut -d ',' -f 2-${nu}`
 echo $spincpu
 fio --ioengine=libaio --runtime=30s --numjobs=${numjobs} --iodepth=${iodepth} --bs=${bs} --size=${size} --rw=${rw} --filename=${filename} --time_based=1 --direct=1 --name=test --group_reporting --cpus_allowed=$spincpu --cpus_allowed_policy=split
}
echo 2 > /sys/block/$DEV_NODE/queue/rq_affinity
sleep 5
RunFio 10 128 4k randwrite 1024g $DEV_NAME

Modify the script parameters for your environment.
- Replace your_device with the actual device name, such as nvme1n1.
- Replace 10 (numjobs), 64 (iodepth), 4k (bs), randwrite (rw), and /dev/your_device in the RunFio 10 64 4k randwrite /dev/your_device line with actual values.
- The numjobs value must not exceed the number of CPU cores. Query the CPU core count:
```
cat /proc/cpuinfo |grep "processor" |wc -l
```
Test the performance of the ESSD:
```
sudo sh test100w.sh
```
- In the output, IOPS=*** shows the ESSD IOPS.
- If the following output appears, the test target contains partitions or file systems. The fio script stops to protect data. Use an empty data disk for testing.
```
[[ecs-user@ecs tmp]$ sudo sh test100w.sh
The /dev/vdb device contains a partition table or a file system. The fio script is stopped.
```
  Warning
  Running fio on a disk with partitions, file systems, or data may corrupt file systems and cause data loss. If your data disk has partitions and file systems, create an empty data disk for testing.
  
  Create a pay-as-you-go disk with the same specifications as your data disk and attach it to the instance. See Create a data disk.
  
  After testing, detach and release the pay-as-you-go disk as needed. See Detach data disk and Release a cloud disk.

Script details: test100w.sh

The following command sets rq_affinity to 2:

echo 2 > /sys/block/your_device/queue/rq_affinity

Value of rq_affinity	Description
1	Delivers I/O completion events to the vCPU group that submitted the I/O requests. Under multi-thread concurrency, events may queue on a single vCPU, creating a bottleneck.
2	Delivers I/O completion events to the exact vCPU that submitted each request. Each vCPU can deliver maximum performance under multi-thread concurrency.

The following command uses jobs to bind queues to different CPU cores:

fio -ioengine=libaio -runtime=30s -numjobs=${numjobs} -iodepth=${iodepth} -bs=${bs} -rw=${rw} -filename=${filename} -time_based=1 -direct=1 -name=test -group_reporting -cpus_allowed=$spincpu -cpus_allowed_policy=split

Note

In normal mode, a device has one request queue, which becomes a bottleneck under concurrent I/O. Multi-queue mode enables multiple request queues. For example, with four I/O threads, bind each thread to a CPU core mapped to a different request queue to maximize storage performance.

Parameter	Description	Example value
`numjobs`	The number of I/O threads.	10
`/dev/your_device`	The device name of the ESSD.	/dev/nvme1n1
`cpus_allowed_policy`	The fio parameter for binding vCPUs. Use `cpus_allowed_policy` and `cpus_allowed` together to bind vCPUs.	split

The command runs jobs to bind queues with different IDs to different CPU cores. To view the CPU core bound to each queue:

Run ls /sys/block/your_device/mq/. Replace your_device with the actual device name (e.g., nvme1n1). This returns queue IDs for ESSDs with /dev/vd* device names.
Run cat /sys/block/your_device/mq/cpu_list. Replace your_device with the actual device name (e.g., nvme1n1). This returns the CPU core ID bound to each queue for ESSDs with /dev/vd* device names.

Prerequisites

Procedure

Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, or CentOS 6 or later

Debian 9 or later, or Ubuntu 14 or later

Script details: test100w.sh