This topic describes how to use Velero with restic integrated to migrate applications and persistent volumes (PVs) from a self-built Kubernetes cluster to a Container Service for Kubernetes cluster. You can also use the same method to migrate applications and PVs from a Kubernetes cluster managed by another cloud vendor to a Container Service for Kubernetes cluster.

Background information

In this example, you migrate a WordPress application from a self-built Kubernetes cluster to a Container Service for Kubernetes cluster. The WordPress application consists of the wordpress and mysql components. Each component uses a separate Network File System (NFS) PV to store application data. The WordPress application is exposed through a NodePort service.

Procedure

  1. Step 1: Migrate container images to Container Registry
  2. Step 2: Deploy MinIO in the Container Service for Kubernetes cluster
  3. Step 3: Deploy a WordPress application and Velero in the self-built Kubernetes cluster
  4. Step 4: Deploy Velero in the Container Service for Kubernetes cluster and restore the WordPress application
  5. Step 5: Check the running status of the WordPress application

Step 1: Migrate container images to Container Registry

Generally, a self-built Kubernetes cluster resides in a local IDC and container images are stored in a self-built image repository. Before migrating applications, you must migrate the container images to Container Registry. For more information, see Migrate container images.

The WordPress application uses the following container images: 
registry.api.paas.com:5000/admin/wordpress:latest
registry.api.paas.com:5000/admin/mysql:8
After the migration, the corresponding images in Container Registry are as follows: 
registry.cn-hangzhou.aliyuncs.com/ack-migration/wordpress:latest
registry.cn-hangzhou.aliyuncs.com/ack-migration/mysql:8

Step 2: Deploy MinIO in the Container Service for Kubernetes cluster

MinIO is an object storage service. When you migrate the WordPress application, you can use MinIO to temporarily store data of the PVs used by the WordPress application.

  1. Create a namespace for MinIO. 
    kubectl create ns minio
  2. Set a username and password for logging on to MinIO. Replace MINIO_ACCESS_KEY_VALUE and MINIO_SECRET_KEY_VALUE in the minio.yaml file with the user name and password respectively. Then, deploy MinIO in the Container Service for Kubernetes cluster. 
    export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=admin12345
export BUCKET_NAME=velero
sed -i "s/MINIO_ACCESS_KEY_VALUE/$MINIO_ACCESS_KEY/g" minio.yaml
sed -i "s/MINIO_SECRET_KEY_VALUE/$MINIO_SECRET_KEY/g" minio.yaml
sed -i "s/BUCKET/$BUCKET_NAME/g" minio.yaml
kubectl apply -f minio.yaml
    A 20 GB standard solid-state drive (SSD) disk is attached to the MinIO application for storing the data of PVs. The MinIO application is exposed through a LoadBalancer service. A bucket named velero is created in MinIO.
    The sample code in the minio.yaml file is as follows: 
    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: storage-pvc
  namespace: minio-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: alicloud-disk-ssd
  resources:
    requests:
      storage: 20Gi

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  namespace: minio-test
  name: minio
  labels:
    component: minio
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        component: minio
    spec:
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: storage-pvc
      - name: config
        emptyDir: {}
      containers:
      - name: minio
        image: minio/minio:latest
        imagePullPolicy: IfNotPresent
        args:
        - server
        - /storage
        - --config-dir=/config
        env:
        - name: MINIO_ACCESS_KEY
          value: MINIO_ACCESS_KEY_VALUE
        - name: MINIO_SECRET_KEY
          value: MINIO_SECRET_KEY_VALUE
        ports:
        - containerPort: 9000
        volumeMounts:
        - name: storage
          mountPath: "/storage"
        - name: config
          mountPath: "/config"

---
apiVersion: v1
kind: Service
metadata:
  namespace: minio-test
  name: minio
  labels:
    component: minio
spec:
  # ClusterIP is recommended for production environments.
  # Change to NodePort if needed per documentation,
  # but only if you run MinIO in a test/trial environment, for example with Minikube.
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    component: minio

 ---
 apiVersion: batch/v1
kind: Job
metadata:
  namespace: minio-test
  name: minio-setup
  labels:
    component: minio
spec:
  template:
    metadata:
      name: minio-setup
    spec:
      restartPolicy: OnFailure
      volumes:
      - name: config
        emptyDir: {}
      containers:
      - name: mc
        image: minio/mc:latest
        imagePullPolicy: IfNotPresent
        command:
        - /bin/sh
        - -c
        - "mc --config-dir=/config config host add BUCKET http://minio:9000 MINIO_ACCESS_KEY_VALUE MINIO_SECRET_KEY_VALUE  && mc --config-dir=/config mb -p BUCKET/BUCKET"
        volumeMounts:
        - name: config
          mountPath: "/config"
  3. Go to the Container Service console. Choose Ingresses and Load Balancing > Services in the left-side navigation pane. On the Services page, select the target cluster and namespace.
    In the service list that appears, you can find the deployed MinIO service.
  4. Click the public endpoint of the MinIO service to access MinIO Browser by using the IP address and port number.

Step 3: Deploy a WordPress application and Velero in the self-built Kubernetes cluster

  1. Deploy a WordPress application in the self-built Kubernetes cluster.
    Note In this example, you create a WordPress application in the self-built Kubernetes cluster and migrate the application to a Container Service for Kubernetes cluster. If you want to migrate an existing application, skip this step.
    The WordPress application consists of the wordpress and mysql components. Each component uses a separate NFS PV to store application data. The WordPress application is exposed through a NodePort service. The sample code in the YAML file for deploying the WordPress application is as follows: 
    # 1. Create an NFS StorageClass.
$ cat nfs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs
provisioner: helm.default/nfs
reclaimPolicy: Delete
$ kubectl apply -f  nfs-sc.yaml

# 2. Create a secret to store the password of the mysql component. Assume that the password is mysql. You can run the echo -n "mysql" |base64 command to obtain the Base64-encoded string of the password.
$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysql
type: Opaque
data:
  password: bXlzcWw=
$ kubectl apply -f secret.yaml

# 3. Create a service, a persistent volume claim (PVC), and a deployment for the mysql component.
$ cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  type: ClusterIP
  ports:
    - port: 3306
  selector:
    app: mysql
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-volumeclaim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      securityContext:
        runAsUser: 999
        runAsGroup: 999
        fsGroup: 999
      containers:
        - image: registry.api.paas.com:5000/admin/mysql:8
          name: mysql
          args:
            - "--default-authentication-plugin=mysql_native_password"
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-volumeclaim
 $ kubectl apply -f mysql.yaml

 # 4. Create a PVC, a deployment, and a service for the wordpress component.
 $ cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: wordpress
  name: wordpress
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 31570
  selector:
    app: wordpress
  type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wordpress-volumeclaim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      containers:
        - image: registry.api.paas.com:5000/admin/wordpress
          name: wordpress
          env:
          - name: WORDPRESS_DB_HOST
            value: mysql:3306
          - name: WORDPRESS_DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysql
                key: password
          ports:
            - containerPort: 80
              name: wordpress
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
      volumes:
        - name: wordpress-persistent-storage
          persistentVolumeClaim:
            claimName: wordpress-volumeclaim
 $ kubectl apply -f wordpress.yaml
    In the test environment, bind the domain name of the WordPress application to the IP address of the node where the application is deployed in the hosts file of your operating system. Then, access the WordPress application through http://wordpress.myk8s.paas.com:31570/. Make sure that the WordPress application is accessible.
  2. Deploy Velero in the self-built Kubernetes cluster.
    1. Download Velero, install it, and verify the installation by checking the Velero version.
      The installation package used in this example is velero-v1.2.0-linux-amd64.tar.gz. 
      $ tar xvf velero-v1.2.0-linux-amd64.tar.gz
$ mv velero-v1.2.0-linux-amd64/velero /usr/bin/velero
$ velero version
Client:
  Version: v1.2.0
  Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
Server:
  Version:
    2. Add the access information of MinIO to the credentials-velero file. 
      [default]
aws_access_key_id=<YourAccessKeyId>
aws_secret_access_key=<YourAccessKeySecret>
      Note In the preceding code, replace YourAccessKeyId and YourAccessKeySecret with the username and password for logging on to MinIO.
    3. Replace MINIO_SERVER_ADDRESS with the actual address of the MinIO server in the following command and run the command to deploy Velero. 
      velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:v1.2.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://MINIO_SERVER_ADDRESS:9000 --use-restic --plugins registry.cn-hangzhou.aliyuncs.com/acs/velero-plugin-for-aws:v1.0.0 --wait
    4. Check the running status of pods. 
      kubectl -n velero get po
NAME                      READY   STATUS    RESTARTS   AGE
restic-fqwsc              1/1     Running   0          41s
restic-kfzqt              1/1     Running   0          41s
restic-klxhc              1/1     Running   0          41s
restic-ql2kr              1/1     Running   0          41s
restic-qrsrn              1/1     Running   0          41s
restic-srjmm              1/1     Running   0          41s
velero-67b975f5cb-68nj4   1/1     Running   0          41s
  3. Back up the WordPress application.
    • If you want to back up only the WordPress application without PVs, run the following commands: 
      $ velero backup create wordpress-backup-without-pv --include-namespaces wordpress
Backup request "wordpress-backup-without-pv" submitted successfully.
Run `velero backup describe wordpress-backup-without-pv` or `velero backup logs wordpress-backup-without-pv` for more details.

$ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    • If you want to back up the WordPress application with PVs, run the following commands: 
      # Annotate the pods to which PVs are attached. Assume that the WordPress application runs in two pods, namely, wordpress-7cf5849f47-mbvx4 and mysql-74dddbdcc8-h2tls.
# The PV attached to the wordpress-7cf5849f47-mbvx4 pod is mysql-persistent-storage. The PV attached to the mysql-74dddbdcc8-h2tls pod is wordpress-persistent-storage. Then, the commands for annotating the pods are as follows:
$ kubectl -n wordpress annotate pod/wordpress-7cf5849f47-mbvx4 backup.velero.io/backup-volumes=wordpress-persistent-storage
pod/wordpress-7cf5849f47-mbvx4 annotated
$ kubectl -n wordpress annotate pod/mysql-74dddbdcc8-h2tls backup.velero.io/backup-volumes=mysql-persistent-storage
pod/mysql-74dddbdcc8-h2tls annotated

# Back up the WordPress application.
$ velero backup create wordpress-backup-with-pv --include-namespaces wordpress
Backup request "wordpress-backup-with-pv" submitted successfully.
Run `velero backup describe wordpress-backup-with-pv` or `velero backup logs wordpress-backup-with-pv` for more details.
$ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
    You can find the backup files on the MinIO server.

Step 4: Deploy Velero in the Container Service for Kubernetes cluster and restore the WordPress application

  1. Deploy Velero in the Container Service for Kubernetes cluster.
    The method for deploying Velero in the Container Service for Kubernetes cluster is similar to that in the self-built Kubernetes cluster. For more information, see Deploy Velero in the self-built Kubernetes cluster.
    After you deploy Velero in the Container Service for Kubernetes cluster, run the following command to check the status of backups: 
    $ velero backup get
NAME                          STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
wordpress-backup-with-pv      Completed   2019-12-12 14:23:40 +0800 CST   29d       default            <none>
wordpress-backup-without-pv   Completed   2019-12-12 14:08:24 +0800 CST   29d       default            <none>
  2. Restore the WordPress application to the Container Service for Kubernetes cluster.
    In the self-built Kubernetes cluster, the WordPress application uses NFS PVs. Accordingly, you can configure Apsara File Storage NAS volumes for the WordPress application in the Container Service for Kubernetes cluster. In this example, you create an NFS StorageClass for the WordPress application but use Alibaba Cloud SSD disks as back-end storage.

    In this example, the Container Service for Kubernetes cluster uses the Container Storage Interface (CSI) plug-in. For more information, see Use dynamic disk volumes.

    1. Create a StorageClass.
      If you restore the WordPress application without PVs, skip this step. 
      $ cat nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: nfs
provisioner: diskplugin.csi.alibabacloud.com
parameters:
    type: cloud_ssd
reclaimPolicy: Retain

$ kubectl apply -f nfs.yaml
storageclass.storage.k8s.io/nfs created
    2. Restore the WordPress application. 
      $ velero restore create --from-backup wordpress-backup-with-pv
$ velero restore get
NAME                                      BACKUP                     STATUS       WARNINGS   ERRORS   CREATED                         SELECTOR
wordpress-backup-with-pv-20191212152745   wordpress-backup-with-pv   InProgress   0          0        2019-12-12 15:27:45 +0800 CST   <none>
$ velero restore get
      If you check the running status of the WordPress application now, errors are returned, indicating that the container images cannot be pulled. 
      $ kubectl -n wordpress get po
NAME                         READY   STATUS         RESTARTS   AGE
mysql-669b4666cd-trsnz       0/1     ErrImagePull   0          19m
mysql-74dddbdcc8-h2tls       0/1     Init:0/1       0          19m
wordpress-7cf5849f47-mbvx4   0/1     Init:0/1       0          19m
wordpress-bb5d74d95-xcjxw    0/1     ErrImagePull   0          19m
    3. Set the image fields in the wordpress and mysql deployments to the URLs of the container images migrated to Container Registry. For more information, see Step 1: Migrate container images to Container Registry. 
      # Modify the image URLs in the deployments.
$ kubectl -n wordpress edit deployment mysql
$ kubectl -n wordpress edit deployment wordpress

Step 5: Check the running status of the WordPress application

Run the following command to check the running status of the WordPress application: 
$ kubectl -n wordpress get po
NAME                         READY   STATUS    RESTARTS   AGE
mysql-678b5d8499-vckfd       1/1     Running   0          100s
wordpress-8566f5f7d8-7shk6   1/1     Running   0          3m18s

In the test environment, bind the domain name of the WordPress application to the IP address of the node where the application is deployed in the hosts file of your operating system. Then, access the WordPress application through http://wordpress.myk8s.paas.com:31570/. Make sure that the WordPress application is accessible.