How to test block storage performance on a Linux instance using FIO - Elastic Compute Service

Benchmarking your block storage helps you understand its capabilities and optimize it for the best performance. Testing a raw disk provides the most accurate performance metrics. This topic describes how to use the open-source tool Fio on a Linux system to benchmark key performance metrics of a raw disk, including IOPS, throughput, and latency.

Note

Fio (Flexible I/O Tester) is a powerful, open-source I/O testing tool used to benchmark storage devices with various workloads, such as random or sequential reads and writes.

Procedure

Important

Testing a raw disk provides the most accurate block storage performance results. However, running Fio directly on a block storage device that contains partitions, a file system, or other data will cause file system errors and data loss. Before you begin, create a snapshot to back up your data. For more information, see Create a snapshot manually.

We strongly recommend that you do not test the system disk or a data disk that contains data to avoid data loss. For testing, we recommend that you use a newly created, uninitialized, and empty data disk.
The performance test results are obtained in a test environment and are for reference only. In a production environment, the performance of a cloud disk may vary due to factors such as network conditions and concurrent access. Actual performance will vary.
After the test is complete, handle the new cloud disk as follows:
- If you want to keep the disk: Do not use it directly. You must re-initialize the data disk.
- If you no longer need the disk: Detach the data disk and then release the cloud disk.

This topic uses the Alibaba Cloud Linux 3.2104 LTS 64-bit public image as an example. The steps may vary based on your operating system.

Connect to an ECS instance.

For more information, see Log on to a Linux instance using Workbench.

Run the following command to find the device names of your block storage.

sudo fdisk -lu

In the command output, /dev/vda is the system disk, and /dev/vdb and /dev/vdc are data disks. /dev/vdc does not have a partition.

[ecs-a                          ]$ sudo fdisk -lu
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F51132A7-67B1-4650-806D-FD0DE6E1210C
Device      Start      End  Sectors  Size Type
/dev/vda1    2048     6143     4096    2M BIOS boot
/dev/vda2    6144   415743   409600  200M EFI System
/dev/vda3  415744 83886046 83470303 39.8G Linux filesystem
Disk /dev/vdb: 30 GiB, 32212254720 bytes, 62914560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C36DF120-8650-4188-8043-AEF9C85F31EF
Device     Start      End  Sectors Size Type
/dev/vdb1   2048 62912511 62910464  30G Linux filesystem
Disk /dev/vdc: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

As shown in the command output, the instance has three block storage devices: the system disk /dev/vda, and the data disks /dev/vdb and /dev/vdc.

Run the following command to check whether the block storage devices have partitions or a file system.
```
sudo blkid
```
```
[ecs-a                          ]$ sudo blkid
/dev/vdb1: UUID="9c32c24f-d2b8-4aa8-8" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="4bd66635-f5f4-4dc0-9bdd-664fd5b8d2fb"
/dev/vda2: SEC_TYPE="msdos" UUID="7E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="82a50cd6-9899-41eb-91fe-7027bf257086"
/dev/vda3: LABEL="root" UUID="beef9d8d-ba84-46d9-8" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="e0d4fa20-912d-4e86-943e-6b0368xxxxxx"
/dev/vda1: PARTUUID="d083a7cd-a7ea-4898-89d5-8e1510bed584"
```
The output shows that the devices /dev/vda and /dev/vdb have partitions and file systems. No information about /dev/vdc is returned, which indicates that /dev/vdc has no partition or file system.
Warning
If a device that you want to test contains partitions, a file system, or data, running Fio directly on the device will cause file system errors and data loss. If your data disk contains partitions and a file system, we recommend that you create a new, empty data disk for testing.
1. You can create a pay-as-you-go cloud disk with the same configurations and attach it to the instance for testing. For more information, see Create a data disk.
2. After the test is complete, you can release the instance on demand. For more information, see Release an instance.
Before you benchmark the block storage performance, back up the data on the device you plan to test to prevent data loss. For more information, see Create a snapshot manually.
Note
You are charged for using snapshots. For more information, see Snapshot billing.
Run the following command to install the libaio library and the Fio tool. Choose the command based on your operating system.
Alibaba Cloud Linux 2/3, CentOS 6 and later
Note
CentOS 6 reached end of life (EOL). In accordance with Linux community rules, all content was removed from the following CentOS 6 repository address: http://mirror.centos.org/centos-6/. If you continue to use the default CentOS 6 repository on Alibaba Cloud, an error is reported. To use specific installation packages of CentOS 6, change the CentOS 6 repository address. For more information, see How do I change CentOS 6 repository addresses?
```
sudo yum install libaio libaio-devel fio -y
```
Debian 9 and later, Ubuntu 14 and later
Important
Debian 9 and Debian 10 are end-of-life (EOL). If your ECS instance runs Debian 9 or Debian 10, change the repository address first. For more information, see Change repository addresses for CentOS and Debian after EOL.
```
sudo apt-get update
sudo apt-get install libaio* fio -y
```
Go to the /tmp directory.
```
cd /tmp
```
Run a performance test command. For the specific commands, see the following sections.
- For cloud disks, see Cloud disk performance test commands.
- For local disks, see Local disk performance test commands.
View the test results. The values may vary depending on the cloud disk. The values in the following code blocks are for reference only.

For IOPS-related test results, see the IOPS=*** field. The following sample Fio output shows the results of a random write (Rand_Write) test. In the write row, IOPS=2301 is the key performance metric for random writes:

Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][w=9061KiB/s][w=2265 IOPS][eta 00m:00s]
Rand_Write Testing: (groupid=0, jobs=1): err= 0: pid=15900: Tue Nov 19 11:28:24 2024
  write: IOPS=2301, BW=9206KiB/s (9427kB/s)(1024MiB/113896msec); 0 zone resets
    slat (usec): min=2, max=1718, avg= 5.43, stdev= 7.98
    clat (usec): min=382, max=119809, avg=55606.05, stdev=7040.12
    lat (usec): min=394, max=119812, avg=55611.49, stdev=7039.41
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   51], 10.00th=[   51], 20.00th=[   51],
     | 30.00th=[   51], 40.00th=[   57], 50.00th=[   60], 60.00th=[   61],
     | 70.00th=[   61], 80.00th=[   61], 90.00th=[   61], 95.00th=[   61],
     | 99.00th=[   63], 99.50th=[   65], 99.90th=[   67], 99.95th=[   77],
     | 99.99th=[  101]
   bw (  KiB/s): min= 9000, max=28976, per=100.00%, avg=9213.14, stdev=1317.97, samples=227
   iops        : min= 2250, max= 7244, avg=2303.28, stdev=329.49, samples=227
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.03%, 4=0.45%, 10=0.52%, 20=0.04%, 50=0.82%
  lat (msec)   : 100=98.11%, 250=0.01%

For throughput-related test results, see the BW=*** field, as shown in the following output:

Jobs: 1 (f=1): [W(1)][100.0%][w=85.0MiB/s][w=85 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=17934: Tue Nov 19 11:32:19 2024
  write: IOPS=124, BW=124MiB/s (130MB/s)(1024MiB/8229msec); 0 zone resets
    slat (usec): min=27, max=4026, avg=94.51, stdev=125.97
    clat (msec): min=4, max=1154, avg=513.48, stdev=208.14
     lat (msec): min=4, max=1154, avg=513.57, stdev=208.15
    clat percentiles (msec):
     |  1.00th=[   17],  5.00th=[   75], 10.00th=[   93], 20.00th=[  567],
     | 30.00th=[  584], 40.00th=[  584], 50.00th=[  584], 60.00th=[  584],
     | 70.00th=[  584], 80.00th=[  584], 90.00th=[  592], 95.00th=[  693],
     | 99.00th=[ 1062], 99.50th=[ 1099], 99.90th=[ 1133], 99.95th=[ 1150],
     | 99.99th=[ 1150]
   bw (  KiB/s): min=30720, max=356352, per=96.43%, avg=122881.25, stdev=65548.59, samples=16
   iops        : min=   30, max=  348, avg=120.00, stdev=64.01, samples=16
  lat (msec)   : 10=0.49%, 20=0.68%, 50=2.15%, 100=10.74%, 250=1.86%
  lat (msec)   : 500=3.22%, 750=76.46%, 1000=2.73%, 2000=1.66%
  cpu          : usr=0.89%, sys=0.45%, ctx=1025, majf=0, minf=10
  IO depths    : 1=0.1%, 2=0.2%, 4=0.4%, 8=0.8%, 16=1.6%, 32=3.1%, >=64=93.8%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0 1024 0 0 short=0 0 0 0 dropped=0 0 0 0

For latency-related test results, see the lat (usec) field. In the total latency statistics line lat (usec): min=254, max=169188, avg=432.25, stdev=487.95, the average latency is approximately 432 microseconds.

Jobs: 1 (f=1): [w(1)][100.0%][w=9169KiB/s][w=2292 IOPS][eta 00m:00s]
Rand_Write_Latency_Testing: (groupid=0, jobs=1): err= 0: pid=19530: Tue Nov 19 11:38:31 2024
  write: IOPS=2301, BW=9206KiB/s (9427kB/s)(1024MiB/113903msec); 0 zone resets
    slat (usec): min=2, max=19160, avg= 8.68, stdev=40.53
    clat (usec): min=3, max=169175, avg=423.57, stdev=486.23
     lat (usec): min=254, max=169188, avg=432.25, stdev=487.95
    clat percentiles (usec):
     |  1.00th=[  334],  5.00th=[  343], 10.00th=[  347], 20.00th=[  351],
     | 30.00th=[  355], 40.00th=[  359], 50.00th=[  367], 60.00th=[  371],
     | 70.00th=[  379], 80.00th=[  396], 90.00th=[  465], 95.00th=[  562],
     | 99.00th=[ 1614], 99.50th=[ 2671], 99.90th=[ 5735], 99.95th=[ 6194],
     | 99.99th=[ 7439]
   bw (  KiB/s): min= 5840, max=10416, per=100.00%, avg=9211.26, stdev=495.72, samples=227
   iops        : min= 1460, max= 2604, avg=2302.80, stdev=123.94, samples=227
  lat (usec)   : 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=93.01%, 750=4.06%, 1000=0.97%
  lat (msec)   : 2=1.23%, 4=0.42%, 10=0.30%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  cpu          : usr=0.84%, sys=3.00%, ctx=262469, majf=0, minf=10

Cloud disk performance test commands

Warning

If a device that you want to test contains partitions, a file system, or data, running Fio directly on the device will cause file system errors and data loss. If your data disk contains partitions and a file system, we recommend that you create a new, empty data disk for testing.

You can create a pay-as-you-go cloud disk with the same configurations and attach it to the instance for testing. For more information, see Create a data disk.
After the test is complete:
- To keep the disk, re-initialize the data disk.
- If you no longer need the disk, detach and then release the cloud disk.

The parameter values in the following commands are for reference only. Replace /dev/your_device with the name of the block storage device identified in Step 2. For example, if the device name of the cloud disk to be tested is /dev/vdb, you must replace /dev/your_device with /dev/vdb in the commands. For more information about Fio parameters, see Fio parameters.

Test the random write IOPS of a cloud disk:

sudo fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=Rand_Write_Testing

Test the random read IOPS of a cloud disk:

sudo fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=Rand_Read_Testing

Test the sequential write throughput of a cloud disk:

sudo fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=1024k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=Write_PPS_Testing

Test the sequential read throughput of a cloud disk:

sudo fio -direct=1 -iodepth=64 -rw=read -ioengine=libaio -bs=1024k -size=1G -numjobs=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=Read_PPS_Testing

Test the random write latency of a cloud disk:

sudo fio -direct=1 -iodepth=1 -rw=randwrite -ioengine=libaio -bs=4k -size=1G -numjobs=1 -group_reporting -filename=/dev/your_device -name=Rand_Write_Latency_Testing

Test the random read latency of a cloud disk:

sudo fio -direct=1 -iodepth=1 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=1 -group_reporting -filename=/dev/your_device -name=Rand_Read_Latency_Testing

For detailed instructions on how to test specific types of cloud disks, see Benchmark the IOPS performance of an ESSD.

Local disk performance test commands

Warning

Local disks can only be created when you create an instance. To run the test, create a new instance with the same configurations. For more information, see Create an instance by using the wizard.
After the test is complete, release the instance if you no longer need it.

The following test commands apply to NVMe SSD local disks and SATA HDD local disks. The parameter values in the following commands are for reference only. Replace /dev/your_device with the name of the block storage device identified in Step 2. For example, if the device name of the local disk to be tested is /dev/vdb, you must replace /dev/your_device with /dev/vdb in the commands. For more information about Fio parameters, see Fio parameters.

Test the random write IOPS of a local disk:

sudo fio -direct=1 -iodepth=32 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=4 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the random read IOPS of a local disk:

sudo fio -direct=1 -iodepth=32 -rw=randread -ioengine=libaio -bs=4k -numjobs=4 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the sequential write throughput of a local disk:

sudo fio -direct=1 -iodepth=128 -rw=write -ioengine=libaio -bs=128k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the sequential read throughput of a local disk:

sudo fio -direct=1 -iodepth=128 -rw=read -ioengine=libaio -bs=128k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the random write latency of a local disk:

sudo fio -direct=1 -iodepth=1 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the random read latency of a local disk:

sudo fio -direct=1 -iodepth=1 -rw=randread -ioengine=libaio -bs=4k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the sequential write latency of a local disk:

sudo fio -direct=1 -iodepth=1 -rw=write -ioengine=libaio -bs=4k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Test the sequential read latency of a local disk:

sudo fio -direct=1 -iodepth=1 -rw=read -ioengine=libaio -bs=4k -numjobs=1 -time_based=1 -runtime=1000 -group_reporting -filename=/dev/your_device -name=test

Fio parameters

The following table describes the Fio parameters used in the test commands.

Parameter	Description
direct	Specifies whether to use direct I/O. Default value: 1. 1: uses direct I/O, which means the I/O cache is bypassed and data is written directly to the storage device. 0: does not use direct I/O.
iodepth	The I/O queue depth for the test. For example, `-iodepth=128` indicates that Fio can have up to 128 I/O requests in flight.
rw	The read/write policy for the test. Valid values include: randwrite: random write. randread: random read. read: sequential read. write: sequential write. randrw: mixed random read and write.
ioengine	The I/O engine that Fio uses for the test. `libaio` is typically used because it closely resembles daily application patterns. For more options, see the official Fio documentation.
bs	The block size for each I/O unit. Default value: 4 KiB. You can specify separate values for read and write operations in the `read,write` format. Either value can be left empty to use the default.
size	The size of the test file. Fio reads or writes the specified amount of data and then stops, unless limited by other options such as `runtime`. If this parameter is not specified, Fio uses the full size of the given file or device. You can also specify the size as a percentage from 1 to 100. For example, if you specify `size=20%`, Fio uses 20% of the full size of the file or device.
numjobs	The number of concurrent threads for the test. Default value: 1.
runtime	The test duration in seconds. If this parameter is not specified, FIO completes read/write operations on a file of the size specified by size by using I/O blocks of the size specified by bs.
group_reporting	The display mode for test results. If specified, aggregates results by group instead of by job.
filename	The path to the test target, which can be a device name or a file path. In this topic, all Fio tests target an entire disk without a file system (a raw disk). To avoid accidentally destroying data by testing the wrong disk, the placeholder /dev/your_device is used in the example commands. Remember to replace it with your actual device path.
name	A user-defined name for the test job, such as Rand_Write_Testing in the examples.

For more information about the parameters, see the fio(1) man page.

Procedure

Alibaba Cloud Linux 2/3, CentOS 6 and later

Debian 9 and later, Ubuntu 14 and later

Cloud disk performance test commands

Local disk performance test commands

Fio parameters