Use NVM volumes for high-performance temporary storage and data-intensive workloads - Container Service for Kubernetes

In ACK, you can use non-volatile memory as block storage or a file system without application changes. This simplifies using NVM volumes, improving read and write performance by 2 to 10 times compared to SSDs. This topic explains how to use NVM volumes for low-latency temporary storage and data-intensive workloads.

Use case 1: Low-latency temporary reads and writes

This use case includes the following scenarios:

Temporary storage for continuous integration (CI).
Small file traversal and search.
High-throughput logging.
Temporary file reads and writes in serverless scenarios.

FIO read and write performance benchmark

This section benchmarks the performance of two volume types, PMEM-LVM and SSD, by creating a separate application for each.

Deploy a YAML file to declare a PMEM-LVM PVC and an SSD PVC.

YAML for the PMEM-LVM PVC

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-pmem-lvm
mountOptions:
- dax
parameters:
  lvmType: striping
  nodeAffinity: "true"
  volumeType: PMEM
provisioner: localplugin.csi.alibabacloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-lvm-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: csi-pmem-lvm

YAML for the SSD PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ssd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: alicloud-disk-topology

Deploy a YAML file to mount the volumes and run anfio test.

Mount the PMEM-LVM volume and run the FIO test

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-pmem-lvm
  labels:
    app: pmem-lvm
spec:
  selector:
    matchLabels:
      app: pmem-lvm
  template:
    metadata:
      labels:
        app: pmem-lvm
    spec:
      containers:
      - name: fio-test
        image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio
        command: ["sh", "-c"]
        args: ["sleep 10000"]
        volumeMounts:
          - name: pmem
            mountPath: "/data"
      volumes:
        - name: pmem
          persistentVolumeClaim:
            claimName: pmem-lvm-pvc

Mount the SSD volume and run the FIO test

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-ssd
  labels:
    app: ssd
spec:
  selector:
    matchLabels:
      app: ssd
  template:
    metadata:
      labels:
        app: ssd
    spec:
      containers:
      - name: fio-test
        image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio
        command: ["sh", "-c"]
        args: ["sleep 10000"]
        volumeMounts:
          - name: ssd
            mountPath: "/data"
      volumes:
        - name: ssd
          persistentVolumeClaim:
            claimName: ssd-pvc

Usekubectl exec to enter the test container and run anfio test to benchmark the write performance of the PMEM-LVM volume.

mount | grep csi
cd /data
fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k

Expected output:

write: IOPS=92.0k, BW=363MiB/s (381MB/s)(8812MiB/24262msec)
lat (nsec): min=2054, max=95278, avg=10544.00, stdev=1697.17

Usekubectl exec to enter the test container and run anfio test to benchmark the write performance of the SSD volume.

cd /data
fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k

Expected output:

lat (usec): min=20, max=7168, avg=24.76, stdev=13.97
write: IOPS=37.3k, BW=146MiB/s (153MB/s)(8744MiB/60001msec)

Result analysis

Volume type	IOPS	Throughput
PMEM-LVM	92,000	381 MB/s
SSD	37,000	153 MB/s

The results show that the PMEM-LVM volume, which uses non-volatile memory, delivers 2 to 3 times the performance of the SSD cloud disk.

Use case 2: Data-intensive workloads

This section benchmarks the write performance of a MySQL database, comparing a deployment on an SSD cloud disk to one on an NVM volume.

Deploy MySQL on an SSD cloud disk

Create an SSD cloud disk volume.

Run the following command to create an SSD cloud disk volume with Alibaba Cloud CSI.

kubectl apply -f disk-mysql.yaml

The following is an example of thedisk-mysql.yaml template:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-mysql
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: alicloud-disk-topology

Run the following command to view the created SSD volume.

kubectl get pvc disk-mysql

Expected output:

NAME         STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS               AGE
disk-mysql   Bound    d-bp12pcfhp3n931s8****   100Gi      RWO            alicloud-disk-topology   8s

Create a MySQL database instance that uses an SSD volume.

Run the following command to create a MySQL database instance.

kubectl apply -f mysql-normal-ssd.yaml

The mysql-normal-ssd.yaml template is as follows:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql-normal
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      hostNetwork: true
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql
          mountPath: /var/lib/mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: disk-mysql
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
data:
  username: YWRt****
  password: YWRt****

Run the following command to check the created MySQL database instance.

kubectl get pod | grep mysql-normal

Expected output:

NAME                    READY    STATUS   RESTARTS    AGE
mysql-normal-****       1/1      Running   0          10h

Create a MySQL database with an NVM volume

Create a PMEM-LVM volume.

Run the following command to create a PMEM-LVM volume using Alibaba Cloud CSI.

kubectl apply -f csi-pmem-lvm.yaml

The following is a sample csi-pmem-lvm.yaml template:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pmem-mysql
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: csi-pmem-lvm

Run the following command to view the created PMEM-LVM volume.

kubectl get pvc pmem-mysql

Expected output:

NAME         STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS          AGE
pmem-mysql   Bound    d-****        100Gi      RWO            csi-pmem-lvm          10h

Create a MySQL database on the mounted PMEM volume.

Run the following command to create a MySQL database on the mounted PMEM volume.

kubectl apply -f mysql-normal-pmem.yaml

The following is a sample mysql-normal-pmem.yaml template:

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql-normal
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      hostNetwork: true
      nodeName: <NODE name of worker node>
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql
          mountPath: /var/lib/mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: pmem-mysql
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
type: Opaque
data:
  username: YWRt****
  password: YWRt****

Run the following command to view the created MySQL database instance.

kubectl get pod | grep mysql-normal

Expected output:

NAME                    READY    STATUS   RESTARTS    AGE
mysql-normal-****        1/1     Running   0          10h

Database write performance test

Run the following command in the pod to stress-test the database performance.

sysbench /root/sysbench/point_select.lua run --db-driver=mysql --report-interval=1
--mysql-table-engine=innodb --oltp-table-size=10000 --oltp-tables-count=32 --oltp-test-mode=complex 
--time=100 --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-password=admin 
--mysql-db=sbtest --oltp-read-only=on --oltp-simple-ranges=0 --oltp-sum-ranges=0 
--oltp-order-ranges=0 --oltp-distinct-ranges=0  --oltp-dist-type=uniform --warmup-time=300 
--max-time=30 --max-requests=0 --percentile=99 --num-threads=150

Test Results

Volume Type	Insert Throughput (inserts/second)
SSD	49812
PMEM-LVM	122156

The write performance of a database using non-volatile memory is approximately 2.5 times that of one using an SSD.