Container Service for Kubernetes：Migrasi volume NAS yang disediakan secara statis dari FlexVolume ke CSI

Dampak

• PVC akan dibuat ulang selama migrasi, sehingga menyebabkan Pod melakukan restart.

• Restart Pod mengganggu workload yang sedang berjalan. Lakukan migrasi selama jam sepi.

Perbandingan FlexVolume dan CSI

Prasyarat

Instal plugin CSI di kluster Anda sebelum memulai.

1. Buat file bernama csi-plugin.yaml dan csi-provisioner.yaml.

   Lihat file csi-plugin.yaml

      ---
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: csi-admin
        namespace: kube-system
      ---
      kind: ClusterRole
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: alicloud-csi-plugin
      rules:
        - apiGroups: [""]
          resources: ["secrets"]
          verbs: ["get", "create", "list"]
        - apiGroups: [""]
          resources: ["persistentvolumes"]
          verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
        - apiGroups: [""]
          resources: ["persistentvolumeclaims"]
          verbs: ["get", "list", "watch", "update"]
        - apiGroups: [""]
          resources: ["persistentvolumeclaims/status"]
          verbs: ["get", "list", "watch", "update", "patch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["storageclasses"]
          verbs: ["get", "list", "watch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["csinodes"]
          verbs: ["get", "list", "watch"]
        - apiGroups: [""]
          resources: ["events"]
          verbs: ["get", "list", "watch", "create", "update", "patch"]
        - apiGroups: [""]
          resources: ["endpoints"]
          verbs: ["get", "watch", "list", "delete", "update", "create"]
        - apiGroups: [""]
          resources: ["configmaps"]
          verbs: ["get", "watch", "list", "delete", "update", "create"]
        - apiGroups: [""]
          resources: ["nodes"]
          verbs: ["get", "list", "watch", "update"]
        - apiGroups: ["csi.storage.k8s.io"]
          resources: ["csinodeinfos"]
          verbs: ["get", "list", "watch"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["volumeattachments"]
          verbs: ["get", "list", "watch", "update", "patch"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotclasses"]
          verbs: ["get", "list", "watch", "create"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotcontents"]
          verbs: ["create", "get", "list", "watch", "update", "delete"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshots"]
          verbs: ["get", "list", "watch", "update", "create"]
        - apiGroups: ["apiextensions.k8s.io"]
          resources: ["customresourcedefinitions"]
          verbs: ["create", "list", "watch", "delete", "get", "update", "patch"]
        - apiGroups: ["coordination.k8s.io"]
          resources: ["leases"]
          verbs: ["get", "create", "list", "watch", "delete", "update"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshotcontents/status"]
          verbs: ["update"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["volumeattachments/status"]
          verbs: ["patch"]
        - apiGroups: ["snapshot.storage.k8s.io"]
          resources: ["volumesnapshots/status"]
          verbs: ["update"]
        - apiGroups: ["storage.k8s.io"]
          resources: ["storageclasses"]
          verbs: ["get", "list", "watch"]
        - apiGroups: [""]
          resources: ["namespaces"]
          verbs: ["get", "list"]
        - apiGroups: [""]
          resources: ["pods","pods/exec"]
          verbs: ["create", "delete", "get", "post", "list", "watch", "patch", "update"]
        - apiGroups: ["storage.alibabacloud.com"]
          resources: ["rules"]
          verbs: ["get"]
        - apiGroups: ["storage.alibabacloud.com"]
          resources: ["containernetworkfilesystems"]
          verbs: ["get","list", "watch"]
      ---
      kind: ClusterRoleBinding
      apiVersion: rbac.authorization.k8s.io/v1
      metadata:
        name: alicloud-csi-plugin
      subjects:
        - kind: ServiceAccount
          name: csi-admin
          namespace: kube-system
      roleRef:
        kind: ClusterRole
        name: alicloud-csi-plugin
        apiGroup: rbac.authorization.k8s.io
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: diskplugin.csi.alibabacloud.com
      spec:
        attachRequired: true
        podInfoOnMount: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: nasplugin.csi.alibabacloud.com
      spec:
        attachRequired: false
        podInfoOnMount: true
      ---
      apiVersion: storage.k8s.io/v1
      kind: CSIDriver
      metadata:
        name: ossplugin.csi.alibabacloud.com
      spec:
        attachRequired: false
        podInfoOnMount: true
      ---
      kind: DaemonSet
      apiVersion: apps/v1
      metadata:
        name: csi-plugin
        namespace: kube-system
      spec:
        selector:
          matchLabels:
            app: csi-plugin
        template:
          metadata:
            labels:
              app: csi-plugin
          spec:
            tolerations:
              - operator: Exists
            affinity:
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
            nodeSelector:
              kubernetes.io/os: linux
            serviceAccount: csi-admin
            priorityClassName: system-node-critical
            hostNetwork: true
            hostPID: true
            dnsPolicy: ClusterFirst
            containers:
              - name: disk-driver-registrar
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/diskplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet
                  - name: registration-dir
                    mountPath: /registration
              - name: nas-driver-registrar
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                  - name: registration-dir
                    mountPath: /registration
              - name: oss-driver-registrar
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock"
                  - "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/ossplugin.csi.alibabacloud.com/csi.sock"
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                  - name: registration-dir
                    mountPath: /registration
              - name: csi-plugin
                securityContext:
                  privileged: true
                  allowPrivilegeEscalation: true
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.24.6-55c95dd-aliyun
                args:
                  - "--endpoint=$(CSI_ENDPOINT)"
                  - "--v=2"
                  - "--driver=oss,nas,disk"
                env:
                  - name: KUBE_NODE_NAME
                    valueFrom:
                      fieldRef:
                        apiVersion: v1
                        fieldPath: spec.nodeName
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock
                  - name: MAX_VOLUMES_PERNODE
                    value: "15"
                  - name: SERVICE_TYPE
                    value: "plugin"
                resources:
                  requests:
                    cpu: 100m
                    memory: 128Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                    scheme: HTTP
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                ports:
                  - name: healthz
                    containerPort: 11260
                volumeMounts:
                  - name: kubelet-dir
                    mountPath: /var/lib/kubelet/
                    mountPropagation: "Bidirectional"
                  - name: etc
                    mountPath: /host/etc
                  - name: host-log
                    mountPath: /var/log/
                  - name: ossconnectordir
                    mountPath: /host/usr/
                  - name: container-dir
                    mountPath: /var/lib/container
                    mountPropagation: "Bidirectional"
                  - name: host-dev
                    mountPath: /dev
                    mountPropagation: "HostToContainer"
                  - mountPath: /var/addon
                    name: addon-token
                    readOnly: true
                  - mountPath: /host/var/run/
                    name: fuse-metrics-dir
            volumes:
              - name: fuse-metrics-dir
                hostPath:
                  path: /var/run/
                  type: DirectoryOrCreate
              - name: registration-dir
                hostPath:
                  path: /var/lib/kubelet/plugins_registry
                  type: DirectoryOrCreate
              - name: container-dir
                hostPath:
                  path: /var/lib/container
                  type: DirectoryOrCreate
              - name: kubelet-dir
                hostPath:
                  path: /var/lib/kubelet
                  type: Directory
              - name: host-dev
                hostPath:
                  path: /dev
              - name: host-log
                hostPath:
                  path: /var/log/
              - name: etc
                hostPath:
                  path: /etc
              - name: ossconnectordir
                hostPath:
                  path: /usr/
              - name: addon-token
                secret:
                  defaultMode: 420
                  optional: true
                  items:
                  - key: addon.token.config
                    path: token-config
                  secretName: addon.csi.token
        updateStrategy:
          rollingUpdate:
            maxUnavailable: 30%
          type: RollingUpdate

   Lihat file csi-provisioner.yaml

      ---
      kind: Deployment
      apiVersion: apps/v1
      metadata:
        name: csi-provisioner
        namespace: kube-system
      spec:
        selector:
          matchLabels:
            app: csi-provisioner
        strategy:
          rollingUpdate:
            maxSurge: 0
            maxUnavailable: 1
          type: RollingUpdate
        replicas: 2
        template:
          metadata:
            labels:
              app: csi-provisioner
          spec:
            affinity:
              nodeAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 1
                  preference:
                    matchExpressions:
                    - key: node-role.kubernetes.io/master
                      operator: Exists
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                  - matchExpressions:
                    - key: type
                      operator: NotIn
                      values:
                      - virtual-kubelet
              podAntiAffinity:
                preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  podAffinityTerm:
                    labelSelector:
                      matchExpressions:
                      - key: app
                        operator: In
                        values:
                        - csi-provisioner
                    topologyKey: kubernetes.io/hostname
            tolerations:
            - effect: NoSchedule
              operator: Exists
              key: node-role.kubernetes.io/master
            - effect: NoSchedule
              operator: Exists
              key: node.cloudprovider.kubernetes.io/uninitialized
            serviceAccount: csi-admin
            hostPID: true
            priorityClassName: system-node-critical
            containers:
              - name: external-disk-provisioner
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--csi-address=$(ADDRESS)"
                  - "--feature-gates=Topology=True"
                  - "--volume-name-prefix=disk"
                  - "--strict-topology=true"
                  - "--timeout=150s"
                  - "--leader-election=true"
                  - "--retry-interval-start=500ms"
                  - "--extra-create-metadata=true"
                  - "--default-fstype=ext4"
                  - "--v=5"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-disk-attacher
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-attacher:v3.3-72dd428b-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election=true"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-disk-resizer
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 8Gi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
              - name: external-nas-provisioner
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--csi-address=$(ADDRESS)"
                  - "--volume-name-prefix=nas"
                  - "--timeout=150s"
                  - "--leader-election=true"
                  - "--retry-interval-start=500ms"
                  - "--default-fstype=nfs"
                  - "--v=5"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
              - name: external-nas-resizer
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 8Gi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election"
                env:
                  - name: ADDRESS
                    value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
                volumeMounts:
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
              - name: external-oss-provisioner
                args:
                  - --csi-address=$(ADDRESS)
                  - --volume-name-prefix=oss
                  - --timeout=150s
                  - --leader-election=true
                  - --retry-interval-start=500ms
                  - --default-fstype=ossfs
                  - --v=5
                env:
                - name: ADDRESS
                  value: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com/csi.sock
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
                resources:
                  limits:
                    cpu: 500m
                    memory: 1Gi
                  requests:
                    cpu: 10m
                    memory: 16Mi
                volumeMounts:
                - mountPath: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com
                  name: oss-provisioner-dir
              - name: external-csi-snapshotter
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-snapshotter:v4.0.0-a230d5b3-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--csi-address=$(ADDRESS)"
                  - "--leader-election=true"
                  - "--extra-create-metadata=true"
                env:
                  - name: ADDRESS
                    value: /csi/csi.sock
                volumeMounts:
                  - name: disk-provisioner-dir
                    mountPath: /csi
              - name: external-snapshot-controller
                image: registry.cn-hangzhou.aliyuncs.com/acs/snapshot-controller:v4.0.0-a230d5b3-aliyun
                resources:
                  requests:
                    cpu: 10m
                    memory: 16Mi
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                args:
                  - "--v=5"
                  - "--leader-election=true"
              - name: csi-provisioner
                securityContext:
                  privileged: true
                image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.24.6-55c95dd-aliyun
                args:
                  - "--endpoint=$(CSI_ENDPOINT)"
                  - "--v=2"
                  - "--driver=nas,disk,oss"
                env:
                  - name: CSI_ENDPOINT
                    value: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock
                  - name: MAX_VOLUMES_PERNODE
                    value: "15"
                  - name: SERVICE_TYPE
                    value: "provisioner"
                  - name: "CLUSTER_ID"
                    value: "CLUSTER_ID"
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                    scheme: HTTP
                  initialDelaySeconds: 10
                  periodSeconds: 30
                  timeoutSeconds: 5
                  failureThreshold: 5
                readinessProbe:
                  httpGet:
                    path: /healthz
                    port: healthz
                  initialDelaySeconds: 5
                  periodSeconds: 20
                ports:
                  - name: healthz
                    containerPort: 11270
                volumeMounts:
                  - name: host-log
                    mountPath: /var/log/
                  - name: disk-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
                  - name: nas-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
                  - name: oss-provisioner-dir
                    mountPath: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com
                  - mountPath: /var/addon
                    name: addon-token
                    readOnly: true
                  - mountPath: /mnt
                    mountPropagation: Bidirectional
                    name: host-dev
                  - mountPath: /host/etc
                    name: etc
                resources:
                  limits:
                    cpu: 500m
                    memory: 1024Mi
                  requests:
                    cpu: 100m
                    memory: 128Mi
            volumes:
              - name: disk-provisioner-dir
                emptyDir: {}
              - name: nas-provisioner-dir
                emptyDir: {}
              - name: oss-provisioner-dir
                emptyDir: {}
              - name: host-log
                hostPath:
                  path: /var/log/
              - name: etc
                hostPath:
                  path: /etc
                  type: ""
              - name: host-dev
                hostPath:
                  path: /mnt
                  type: ""
              - name: addon-token
                secret:
                  defaultMode: 420
                  optional: true
                  items:
                  - key: addon.token.config
                    path: token-config
                  secretName: addon.csi.token

2. Deploy csi-plugin dan csi-provisioner di kluster:

      kubectl apply -f csi-plugin.yaml -f csi-provisioner.yaml

3. Verifikasi bahwa CSI sedang berjalan: Jika semua Pod menunjukkan status Running, berarti CSI telah terinstal.

      kubectl get pods -nkube-system | grep csi

      csi-plugin-577mm                              4/4     Running   0          3d20h
      csi-plugin-k9mzt                              4/4     Running   0          41d
      csi-provisioner-6b58f46989-8wwl5              9/9     Running   0          41d
      csi-provisioner-6b58f46989-qzh8l              9/9     Running   0          6d20h

Prosedur

Contoh berikut menunjukkan migrasi untuk workload StatefulSet. Gambar di bawah ini menampilkan alur kerja:

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: kubelet-set
spec:
  selector:
    matchLabels:
      app: kubelet-set
  template:
    metadata:
      labels:
        app: kubelet-set
    spec:
      tolerations:
        - operator: "Exists"
      hostNetwork: true
      hostPID: true
      containers:
        - name: kubelet-set
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: registry.cn-hangzhou.aliyuncs.com/acs/csi-plugin:v1.26.5-56d1e30-aliyun
          imagePullPolicy: "Always"
          env:
          - name: enableADController
            value: "true"
          command: ["sh", "-c"]
          args:
          - echo "Starting kubelet flag set to $enableADController";
            ifFlagTrueNum=`cat /host/etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep enable-controller-attach-detach=$enableADController | grep -v grep | wc -l`;
            echo "ifFlagTrueNum is $ifFlagTrueNum";
            if [ "$ifFlagTrueNum" = "0" ]; then
                curValue="true";
                if [ "$enableADController" = "true" ]; then
                    curValue="false";
                fi;
                sed -i "s/enable-controller-attach-detach=$curValue/enable-controller-attach-detach=$enableADController/" /host/etc/systemd/system/kubelet.service.d/10-kubeadm.conf;
                restartKubelet="true";
                echo "current value is $curValue, change to expect "$enableADController;
            fi;
            if [ "$restartKubelet" = "true" ]; then
                /nsenter --mount=/proc/1/ns/mnt systemctl daemon-reload;
                /nsenter --mount=/proc/1/ns/mnt service kubelet restart;
                echo "restart kubelet";
            fi;
            while true;
            do
                sleep 5;
            done;
          volumeMounts:
          - name: etc
            mountPath: /host/etc
      volumes:
        - name: etc
          hostPath:
            path: /etc

Referensi

• Ikhtisar Storage (CSI)

• Ikhtisar FlexVolume

• Gunakan csi-compatible-controller untuk migrasi dari FlexVolume ke CSI

• API UnInstallClusterAddons

• API InstallClusterAddons

• Versi csi-provisioner