Best practices for data security of disk volumes - Container Service for Kubernetes

If data corruption occurs when you expand an enhanced SSD (ESSD) volume without service interruptions or data loss occurs due to an accidental ESSD deletion, you can use the instant access (IA) snapshot of the ESSD to restore the data in the ESSD. This topic describes how to restore data in an ESSD by using an IA snapshot.

Prerequisites

A Container Service for Kubernetes (ACK) that runs Kubernetes 1.20 or later is created. For more information, see Create an ACK managed cluster.
Log on the Elastic Compute Service (ECS) console and select the region where your cluster is deployed. Make sure that the ECS Snapshot service is enabled. For more information, see Activate ECS Snapshot.
The versions of csi-plugin and csi-provisioner are 1.24.4-7371f039-aliyun or later. For more information about how to update csi-plugin and csi-provisioner, see Install and upgrade the CSI plug-in.

Scenarios

Data corruption occurs when you expand disk volumes without service interruptions

Due to business development, you may have increasing requirements for the capacity and performance of the disks mounted to the application pods. You can expand disk volumes to increase the storage capacity of your application. To ensure data security, you must perform the following steps in sequence: stop the application pods, unmount the disk volumes, and then expand the disk volumes. This method interrupts your business. However, service interruptions are not acceptable in most cases.

To ensure that you can restore the data that is corrupted when you expand ESSD volumes without service interruptions, ACK can automatically create IA snapshots for ESSDs before disk expansion. This allows you to restore the data in ESSDs by using the IA snapshots created by ACK.

Data loss occurs due to accidental ESSD deletions

If the reclaim policy of an ESSD is Delete, after you delete the persistent volume claim (PVC) that is used to mount the ESSD, the related persistent volume (PV) and ESSD are also deleted. When you accidentally deleted a disk, you can use a snapshot of the disk to restore the disk data to the point in time when the snapshot was created. You cannot restore disk data if no snapshot is created. You cannot restore the data that is written into the disk after the snapshot was created.

To ensure that you can restore disk data when you accidentally delete an ESSD, ACK can automatically create IA snapshots for ESSDs before disk deletion. The IA snapshots created by ACK are retained for a specific period of time. This allows you to restore the data in ESSDs by using the IA snapshots created by ACK.

Benefits

If you failed to expand a disk, you can restore the disk data by using an IA snapshot of the disk.
If a disk is accidentally deleted, you can restore the disk data by using an IA snapshot of the disk.
The IA snapshot feature does not affect the disk expansion or deletion operation because IA snapshots can be created within seconds.

Note Only ESSDs support IA snapshots.

Configurations

csi-provisioner configurations

Run the following command to add the following environment variable to the containers parameter of csi-provisioner in the kube-system namespace: VOLUME_DEL_AUTO_SNAP: "true".

kubectl patch deploy csi-provisioner -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"csi-provisioner","env":[{"name":"VOLUME_DEL_AUTO_SNAP","value":"true"}]}]}}}}'

Note ACK can automatically create IA snapshots for ESSDs before disk expansion only when you set VOLUME_DEL_AUTO_SNAP to true.

StorageClass configurations

Create a StorageClass based on the following template:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-datasafe-essd
provisioner: diskplugin.csi.alibabacloud.com
parameters:
  type: cloud_essd
  volumeExpandAutoSnapshot: "forced"       # This setting takes effect only when the type parameter is set to "cloud_essd". 
  volumeDeleteSnapshotRetentionDays: "3"   # This setting takes effect only when the reclaimPolicy parameter is set to "Delete". 
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true


Parameter	Description
volumeExpandAutoSnapshot	Specifies whether to automatically create IA snapshots for the ESSD before the ESSD is expanded. If the ESSD is expanded, the system deletes the IA snapshot that is automatically created before the expansion. If the ESSD failed to be expanded, the system retains the IA snapshot for one day and prints the snapshot ID in the cluster event. Default value: closed. Valid values: forced: cancels the expansion if the system fails to create an IA snapshot. besteffort: generates a warning event and does not cancel the expansion if the system fails to create an IA snapshot. closed: does not create an IA snapshot before the ESSD is expanded.
volumeDeleteSnapshotRetentionDays	The number of days for which the system retains the IA snapshot that is automatically created before the ESSD is expanded. If you leave this parameter empty, the system cannot create IA snapshots. We recommend that you set this parameter.

Create an IA snapshot

In this section, a MySQL application is created and has an ESSD mounted.

Create a file named mysql.yaml based on the following requirements:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: essd-pvc
  namespace: autosnapshot
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: alicloud-datasafe-essd  # Use the StorageClass that you created in the previous section. 
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-pass
  namespace: autosnapshot
type: Opaque
data:
  username: dGVzdDEK
  password: dGVzdDEtdmFsdWUK
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-sts
  namespace: autosnapshot
spec:
  selector:
    matchLabels:
      app: mysql-sts
  serviceName: mysql-sts
  template:
    metadata:
      labels:
        app: mysql-sts
    spec:
      containers:
      - name: mysql-sts
        image: mysql:5.7
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 80
          name: mysql-sts
        volumeMounts:
        - name: mysql
          mountPath: /var/lib/mysql
          subPath: mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: essd-pvc

Run the following command to deploy the MySQL application:
```
kubectl apply -f mysql.yaml
```
Run the following command to query the disk volume that is mounted to the MySQL application:
```
kubectl get pvc -nautosnapshot | grep essd-pvc
```
Expected output:
```
essd-pvc   Bound    d-2zeit7uza22vjya1****   25Gi       RWO            alicloud-datasafe-essd   54s
```
The ID of the mounted ESSD is d-2zeit7uza22vjya1****.
Run the following command to access the container of the MySQL application:
```
kubectl -n autosnapshot exec -it mysql-sts-0 -- /bin/sh
```
Run the following command in a shell of the container to write data into the container:
```
dd if=/dev/urandom of=/var/lib/mysql/mysql/record.txt bs=1M count=1000
```
Run the following command to query the size of the data that is written into the container:
```
ls /var/lib/mysql/mysql -l | grep record
```
Expected output:
```
-rw-r--r-- 1 root  root  1048576000 Nov  8 02:36 record.txt
```

Expand the disk volume

Run the following command to expand the disk volume that is mounted to the MySQL application:

kubectl patch pvc essd-pvc -n autosnapshot -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'

Run the following command to check whether the disk volume is expanded and whether an IA snapshot is created before the expansion:
```
kubectl describe pvc essd-pvc -n autosnapshot
```
Expected output:
```
Capacity:   30Gi
...
Events:
...ControllerExpandVolume:: Snapshot create successful: snapshotName[volume-expand-auto-snapshot-d-2ze9belqefqhdmga****-2022-11-07-21:03:12], sourceId[d-2zeit7uza22vjya1****], snapshotId[s-2ze3pmo5ppaa9stb****]
```
The following list describes some parameters in the preceding output:
- sourceId: the ID of the ESSD that is expanded. The ID is the same as the ID of the ESSD that is mounted by using the essd-pvc PVC.
- snapshotId: the ID of the IA snapshot that is automatically created before the expansion.
  - If the ESSD is expanded, you can search for the IA snapshot by ID in the ECS console. If no snapshot matches the ID, the automatically created IA snapshot is deleted after the ESSD is expanded.
  - If the ESSD fails to be expanded, the IA snapshot is retained. The default retention period is one day.

Delete the disk

Run the following command to reduce the number of replicated pods to 0 for the MySQL application:
```
kubectl scale sts/mysql-sts -n autosnapshot --replicas=0
```
Run the following command to delete the PVC that is used by the MySQL application:
```
kubectl delete pvc essd-pvc -n autosnapshot
```
Run the following command to query the VolumeSnapshot that is created for the IA snapshot:
```
kubectl get volumesnapshot
```
Expected output:
```
d-2zeit7uza22vjya1****-delprotect                   true                                d-2zeit7uza22vjya1****-delprotect-content   30Gi                               d-2zeit7uza22vjya1****-delprotect-content          6s             6s
```
The following list describes some parameters in the preceding output:
- d-2zeit7uza22vjya1****: the ID of the ESSD that is deleted. The ID is the same as the ID of the ESSD that is mounted by using the essd-pvc PVC.
- d-2zeit7uza22vjya1****-delprotect: the name of the VolumeSnapshot that is created for the IA snapshot.
- d-2zeit7uza22vjya1****-delprotect-content: the name of the VolumeSnapshotContent that is created for the IA snapshot.

Use the IA snapshot to restore data after the disk is expanded or deleted

In this section, the IA snapshot created in the previous Delete the disk section is used to restore the disk data after the disk is deleted. You can restore the disk data in a similar way after the disk is expanded.

Note By default, the original VolumeSnapshot created for the IA snapshot belongs to the default namespace. If your application is deployed in a different namespace, you must create a VolumeSnapshot in the namespace for the IA snapshot. In this example, the MySQL application is deployed in the autosnapshot namespace. You must create a VolumeSnapshot in the autosnapshot namespace.

Run the following command to query the value of the snapshotHandle parameter of the VolumeSnapshotContent that is created for the IA snapshot:
```
kubectl get volumesnapshotcontent d-2zeit7uza22vjya1****-delprotect-content -oyaml | grep snapshotHandle
```
Expected output:
```
snapshotHandle: s-2zegw6gmuc866xgc****
```

Create a VolumeSnapshotContent by using the following YAML template. The template specifies the VolumeSnapshot to be created.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: datasafe-volumesnapshotcontent
spec:
  deletionPolicy: Retain
  driver: diskplugin.csi.alibabacloud.com
  source:
    snapshotHandle: s-2zegw6gmuc866xgc****   # Set to the value of the snapshotHandle parameter of the original VolumeSnapshotContent. 
  volumeSnapshotRef:
    name: datasafe-volumesnapshot       # The name of the VolumeSnapshot to be created. 
    namespace: autosnapshot             # The namespace in which the MySQL application is deployed.

Create a VolumeSnapshot in the namespace in which the MySQL application is deployed by using the following YAML template:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: datasafe-volumesnapshot
  namespace: autosnapshot
spec:
  source:
    volumeSnapshotContentName: datasafe-volumesnapshotcontent

Create a PVC by using the following YAML template. The template specifies the VolumeSnapshot to be used.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: essd-pvc       # Set to the name of the PVC used by the MySQL application. 
spec:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: alicloud-datasafe-essd
  resources:
    requests:
      storage: 30Gi
  dataSource:
    name: datasafe-volumesnapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Run the following command to increase the number of replicated pods to 1 for the MySQL application:
```
kubectl scale sts/mysql-sts -n autosnapshot --replicas=1
```
Run the following command to check whether the PVC is mounted to the MySQL application:
```
kubectl describe pvc essd-pvc -n autosnapshot | grep "Used By"
```
Expected output:
```
Used By:     mysql-sts-0
```
Run the following command to access the container of the MySQL application:
```
kubectl -n autosnapshot exec -it mysql-sts-0 -- /bin/sh
```
Run the following command to check whether the data that you previously wrote into the container is restored:
```
ls /var/lib/mysql/mysql -l | grep record
```
Expected output:
```
-rw-r--r-- 1 root  root  1048576000 Nov  8 02:36 record.txt
```
In the output, 1048576000 is returned for the size of the data. The size is the same as the size of the data that you previously wrote into the container. This indicates that the data restoration is successful.