Container Service for Kubernetes (ACK) allows you to use non-volatile memory (NVM) volumes as block storage or file systems without modifications to your applications. This makes it easy to use NVM resources and improves read and write performance by 2 to 10 times compared with standard SSDs. This topic describes how to use NVM volumes in ACK clusters to meet the requirements of data-intensive workloads and scenarios that require low-latency and high-speed reads and writes to temporary storage.
Scenario 1: Low-latency and high-speed temporary storage
The following scenarios require low-latency and high-speed temporary storage:
Temporary storage in continuous integration and continuous delivery (CD/CD)
Small file traversal
High-throughput logging
Temporary file reads and writes in serverless scenarios
Use FIO to test read and write performance
In this topic, two persistent volume claims (PVCs) are created based on AEP PMEM-LVM and SSD to test read and write performance.
Deploy the following YAML file to create two PVCs based on PMEM-LVM and SSD:
YAML template for the PVC of the PMEM-LVM type
allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-pmem-lvm mountOptions: - dax parameters: lvmType: striping nodeAffinity: "true" volumeType: PMEM provisioner: localplugin.csi.alibabacloud.com reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pmem-lvm-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: csi-pmem-lvm
YAML template for the PVC of the SSD type
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ssd-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: alicloud-disk-topology
Deploy the following YAML file to mount the volumes to pods and run FIO tests:
Create an application that has the PMEM-LVM type PVC mounted and run FIO tests
apiVersion: apps/v1 kind: Deployment metadata: name: deployment-pmem-lvm labels: app: pmem-lvm spec: selector: matchLabels: app: pmem-lvm template: metadata: labels: app: pmem-lvm spec: containers: - name: fio-test image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio command: ["sh", "-c"] args: ["sleep 10000"] volumeMounts: - name: pmem mountPath: "/data" volumes: - name: pmem persistentVolumeClaim: claimName: pmem-lvm-pvc
Create an application that has the SSD type PVC mounted and run FIO tests
apiVersion: apps/v1 kind: Deployment metadata: name: deployment-ssd labels: app: ssd spec: selector: matchLabels: app: ssd template: metadata: labels: app: ssd spec: containers: - name: fio-test image: registry.cn-hangzhou.aliyuncs.com/eric-dev/sysbench:fio command: ["sh", "-c"] args: ["sleep 10000"] volumeMounts: - name: ssd mountPath: "/data" volumes: - name: ssd persistentVolumeClaim: claimName: ssd-pvc
Use the
kubectl exec
command to run FIO tests inside containers to test the write performance of the PMEM-LVM volume:mount | grep csi cd /data fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k
Expected output:
write: IOPS=92.0k, BW=363MiB/s (381MB/s)(8812MiB/24262msec) lat (nsec): min=2054, max=95278, avg=10544.00, stdev=1697.17
Use the
kubectl exec
command to run FIO tests inside containers to test the write performance of the SSD volume:cd /data fio -filename=./testfile -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=10G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k
Expected output:
lat (usec): min=20, max=7168, avg=24.76, stdev=13.97 write: IOPS=37.3k, BW=146MiB/s (153MB/s)(8744MiB/60001msec)
The output indicates that the read and write performance of the AEP device is two to three times higher than that of the SSD device.
Volume type
IOPS
Throughput
PMEM-LVM
92000
381 MB/s
SSD
37000
153 MB/s
Scenario 2: Data-intensive workloads
In this topic, a MySQL instance that has an SSD volume mounted and a MySQL instance that has an NVM volume mounted are created. Then, the write performance of the MySQL instance is tested.
Create a MySQL instance that has an SSD volume mounted
Create an SSD volume.
Run the following command to create an SSD volume by using the Container Storage Interface (CSI) driver provided by Alibaba Cloud:
kubectl apply -f disk-mysql.yaml
The following template provides an example of the disk-mysql.yaml file:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: disk-mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: alicloud-disk-topology
Run the following command to query the SSD volume:
kubectl get pvc disk-mysql
Expected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE disk-mysql Bound d-*** 100Gi RWO alicloud-disk-topology 10h
Create a MySQL instance that has the SSD volume mounted.
Run the following command to create a MySQL instance:
kubectl apply -f mysql-normal-ssd.yaml
The following template provides an example of the mysql-normal-ssd.yaml file:
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: mysql-normal labels: app: wordpress spec: selector: matchLabels: app: wordpress tier: mysql strategy: type: Recreate template: metadata: labels: app: wordpress tier: mysql spec: hostNetwork: true containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql mountPath: /var/lib/mysql volumes: - name: mysql persistentVolumeClaim: claimName: disk-mysql --- apiVersion: v1 kind: Secret metadata: name: mysql-pass type: Opaque data: username: YWRt**** password: YWRt****
Run the following command to query the pod that runs the MySQL instance:
kubectl get pod | grep mysql-normal
Expected output:
NAME READY STATUS RESTARTS AGE mysql-normal-**** 1/1 Running 0 10h
Create a MySQL instance that has an NVM volume mounted
Create a PMEM-LVM volume.
Run the following command to create a PMEM-LVM volume by using the CSI driver provided by Alibaba Cloud:
kubectl apply -f csi-pmem-lvm.yaml
The following template provides an example of the csi-pmem-lvm.yaml file:
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pmem-mysql spec: accessModes: - ReadWriteOnce resources: requests: storage: 100Gi storageClassName: csi-pmem-lvm
Run the following command to query the PMEM-LVM volume:
kubectl get pvc pmem-mysql
Expected output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pmem-mysql Bound d-**** 100Gi RWO csi-pmem-lvm 10h
Create a MySQL instance that has the PMEM-LVM volume mounted.
Run the following command to create a MySQL instance that has the PMEM-LVM volume mounted:
kubectl apply -f mysql-normal-pmem.yaml
The following template provides an example of the mysql-normal-pmem.yaml file:
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: mysql-normal labels: app: wordpress spec: selector: matchLabels: app: wordpress tier: mysql strategy: type: Recreate template: metadata: labels: app: wordpress tier: mysql spec: hostNetwork: true nodeName: <NODE name of worker node> containers: - image: mysql:5.6 name: mysql env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-pass key: password ports: - containerPort: 3306 name: mysql volumeMounts: - name: mysql mountPath: /var/lib/mysql volumes: - name: mysql persistentVolumeClaim: claimName: pmem-mysql --- apiVersion: v1 kind: Secret metadata: name: mysql-pass type: Opaque data: username: YWRt**** password: YWRt****
Run the following command to query the pod that runs the MySQL instance:
kubectl get pod | grep mysql-normal
Expected output:
NAME READY STATUS RESTARTS AGE mysql-normal-**** 1/1 Running 0 10h
Test the write performance of the MySQL instance
Run the following command on the pod to test the write performance of the MySQL instance:
sysbench /root/sysbench/point_select.lua run --db-driver=mysql --report-interval=1
--mysql-table-engine=innodb --oltp-table-size=10000 --oltp-tables-count=32 --oltp-test-mode=complex
--time=100 --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-password=admin
--mysql-db=sbtest --oltp-read-only=on --oltp-simple-ranges=0 --oltp-sum-ranges=0
--oltp-order-ranges=0 --oltp-distinct-ranges=0 --oltp-dist-type=uniform --warmup-time=300
--max-time=30 --max-requests=0 --percentile=99 --num-threads=150
Test results
Volume type | Insert IOPS |
SSD | 49812 |
PMEM-LVM | 122156 |
The result indicates that the write performance of the MySQL instance that has an NVM volume mounted is 2.5 times the write performance of the MySQL instance that has an SSD volume mounted.