Use NVM volumes to improve read and write performance - - Alibaba Cloud Documentation Center

Container Service for Kubernetes (ACK) allows you to use non-volatile memory (NVM) volumes as block storage or file systems without modifications to your applications. This makes it easy to use NVM resources and improves read and write performance by 2 to 10 times compared with standard SSDs. This topic describes how to use NVM volumes in ACK clusters to meet the requirements of data-intensive workloads and scenarios that require low-latency and high-speed reads and writes to temporary storage.

Scenario 1: Low-latency and high-speed temporary storage

The following scenarios require low-latency and high-speed temporary storage:

Temporary storage in continuous integration and continuous delivery (CD/CD)
Small file traversal
High-throughput logging
Temporary file reads and writes in serverless scenarios

Use FIO to test read and write performance

In this topic, two persistent volume claims (PVCs) are created based on AEP PMEM-LVM and SSD to test read and write performance.

Deploy the following YAML file to create two PVCs based on PMEM-LVM and SSD:

YAML template for the PVC of the PMEM-LVM type

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-pmem-lvm
mountOptions:
- dax
parameters:
  lvmType: striping
  nodeAffinity: "true"
  volumeType: PMEM
provisioner: localplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-lvm-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: csi-pmem-lvm

YAML template for the PVC of the SSD type

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ssd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: alicloud-disk-topology

Deploy the following YAML file to mount the volumes to pods and run FIO tests:

Create an application that has the PMEM-LVM type PVC mounted and run FIO tests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-pmem-lvm
  labels:
    app: pmem-lvm
spec:
  selector:
    matchLabels:
      app: pmem-lvm
  template:
    metadata:
      labels:
        app: pmem-lvm
    spec:
      containers:
      - name: fio-test
        image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio
        command: ["sh", "-c"]
        args: ["sleep 10000"]
        volumeMounts:
          - name: pmem
            mountPath: "/data"
      volumes:
        - name: pmem
          persistentVolumeClaim:
            claimName: pmem-lvm-pvc

Create an application that has the SSD type PVC mounted and run FIO tests

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-ssd
  labels:
    app: ssd
spec:
  selector:
    matchLabels:
      app: ssd
  template:
    metadata:
      labels:
        app: ssd
    spec:
      containers:
      - name: fio-test
        image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio
        command: ["sh", "-c"]
        args: ["sleep 10000"]
        volumeMounts:
          - name: ssd
            mountPath: "/data"
      volumes:
        - name: ssd
          persistentVolumeClaim:
            claimName: ssd-pvc

Use the kubectl exec command to run FIO tests inside containers to test the write performance of the PMEM-LVM volume:

mount | grep csi
cd /data
fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k

Expected output:

write: IOPS=92.0k, BW=363MiB/s (381MB/s)(8812MiB/24262msec)
lat (nsec): min=2054, max=95278, avg=10544.00, stdev=1697.17

Use the kubectl exec command to run FIO tests inside containers to test the write performance of the SSD volume:

cd /data
fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k

Expected output:

lat (usec): min=20, max=7168, avg=24.76, stdev=13.97
write: IOPS=37.3k, BW=146MiB/s (153MB/s)(8744MiB/60001msec)

The output indicates that the read and write performance of the AEP device is two to three times higher than that of the SSD device.

Volume type	IOPS	Throughput
PMEM-LVM	92000	381 MB/s
SSD	37000	153 MB/s

Scenario 2: Data-intensive workloads

In this topic, a MySQL instance that has an SSD volume mounted and a MySQL instance that has an NVM volume mounted are created. Then, the write performance of the MySQL instance is tested.

Create a MySQL instance that has an SSD volume mounted

Create an SSD volume.

Run the following command to create an SSD volume by using the Container Storage Interface (CSI) driver provided by Alibaba Cloud:

kubectl apply -f disk-mysql.yaml

The following template provides an example of the disk-mysql.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-mysql
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: alicloud-disk-topology

Run the following command to query the SSD volume:

kubectl get pvc disk-mysql

Expected output:

NAME         STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS             AGE
disk-mysql   Bound    d-***       100Gi      RWO            alicloud-disk-topology   10h

Create a MySQL instance that has the SSD volume mounted.

Run the following command to create a MySQL instance:

kubectl apply -f mysql-normal-ssd.yaml

The following template provides an example of the mysql-normal-ssd.yaml file:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql-normal
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      hostNetwork: true
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql
          mountPath: /var/lib/mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: disk-mysql
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
data:
  username: YWRt****
  password: YWRt****

Run the following command to query the pod that runs the MySQL instance:

kubectl get pod | grep mysql-normal

Expected output:

NAME                    READY    STATUS   RESTARTS    AGE
mysql-normal-****       1/1      Running   0          10h

Create a MySQL instance that has an NVM volume mounted

Create a PMEM-LVM volume.

Run the following command to create a PMEM-LVM volume by using the CSI driver provided by Alibaba Cloud:

kubectl apply -f csi-pmem-lvm.yaml

The following template provides an example of the csi-pmem-lvm.yaml file:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-mysql
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: csi-pmem-lvm

Run the following command to query the PMEM-LVM volume:

kubectl get pvc pmem-mysql

Expected output:

NAME         STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS          AGE
pmem-mysql   Bound    d-****        100Gi      RWO            csi-pmem-lvm          10h

Create a MySQL instance that has the PMEM-LVM volume mounted.

Run the following command to create a MySQL instance that has the PMEM-LVM volume mounted:

kubectl apply -f mysql-normal-pmem.yaml

The following template provides an example of the mysql-normal-pmem.yaml file:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql-normal
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      hostNetwork: true
      nodeName: <NODE name of worker node>
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql
          mountPath: /var/lib/mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: pmem-mysql
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
data:
  username: YWRt****
  password: YWRt****

Run the following command to query the pod that runs the MySQL instance:

kubectl get pod | grep mysql-normal

Expected output:

NAME                    READY    STATUS   RESTARTS    AGE
mysql-normal-****        1/1     Running   0          10h

Test the write performance of the MySQL instance

Run the following command on the pod to test the write performance of the MySQL instance:

sysbench /root/sysbench/point_select.lua run --db-driver=mysql --report-interval=1
--mysql-table-engine=innodb --oltp-table-size=10000 --oltp-tables-count=32 --oltp-test-mode=complex 
--time=100 --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-password=admin 
--mysql-db=sbtest --oltp-read-only=on --oltp-simple-ranges=0 --oltp-sum-ranges=0 
--oltp-order-ranges=0 --oltp-distinct-ranges=0  --oltp-dist-type=uniform --warmup-time=300 
--max-time=30 --max-requests=0 --percentile=99 --num-threads=150

Test results

Volume type	Insert IOPS
SSD	49812
PMEM-LVM	122156

The result indicates that the write performance of the MySQL instance that has an NVM volume mounted is 2.5 times the write performance of the MySQL instance that has an SSD volume mounted.