您可以在ACK Windows容器中使用云盘动态存储卷及SMB静态存储资源。本文介绍如何在Windows容器中为Windows容器配置阿里云云盘及基于SMB的文件存储。

前提条件

为Windows容器配置阿里云云盘

阿里云云盘是为阿里云服务器(ECS)提供的一种块存储服务,它具有低时延,高性能,耐用性好及高可靠等特点。

您可以通过以下步骤为Windows容器配置阿里云云盘。

步骤一:创建存储类StorageClass

云盘控制器(Cloud Disk Controller)根据存储类(StorageClass)的参数 fstype的值判断创建的云盘适用于Windows文件系统或Linux文件系统。
  • 当参数fstype的值为ext3ext4xfs时,表示该StorageClass适用于Linux文件系统。
  • 当参数fstype的值为ntfs时,表示该StorageClass适用于Windows文件系统。

默认情况下,若存储插件选择为Flexvolume,则在ACK集群创建时会定义Linux文件系统适用的存储类(StorageClass),而Windows文件系统适用的存储类需要自定义。

  1. 使用以下YAML示例自定义Windows文件系统适用的存储类(StorageClass),并创建storageclass.yaml文件。
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: alicloud-disk-common-windows
    provisioner: alicloud/disk
    parameters:
      type: cloud
      fstype: ntfs
    ---
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: alicloud-disk-efficiency-windows
    provisioner: alicloud/disk
    parameters:
      type: cloud_efficiency
      fstype: ntfs
    ---
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: alicloud-disk-ssd-windows
    provisioner: alicloud/disk
    parameters:
      type: cloud_ssd
      fstype: ntfs
    ---
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: alicloud-disk-essd-windows
    provisioner: alicloud/disk
    parameters:
      type: cloud_essd
      fstype: ntfs
    ---
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: alicloud-disk-available-windows
    provisioner: alicloud/disk
    parameters:
      type: available
      fstype: ntfs
    	  

    以下为一些StorageClass的配置示例说明:
    YAML示例 说明
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: alicloud-disk-ssd-windows
      provisioner: alicloud/disk
      parameters:
        type: cloud_ssd
        fstype: ntfs
    + allowVolumeExpansion: true # useless
    目前Windows不支持盘区扩展。例如,配置的allowVolumeExpansion: true并不能起到预期扩展云盘的作用。
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: alicloud-disk-ssd-windows
      provisioner: alicloud/disk
      parameters:
        type: cloud_ssd
        fstype: ntfs
    + reclaimPolicy: Retain # only unmount
    根据需要使用保留回收策略reclaimPolicy: Retain可以减少由于错误操作导致数据删除的问题。
      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: alicloud-disk-ssd-windows
      provisioner: alicloud/disk
      parameters:
        type: cloud_ssd
        fstype: ntfs
    +   regionId: cn-hangzhou
    +   zoneId: cn-hangzhou-b
    若缺少地域及可用区的定义,云盘可能会挂载失败。使用参数regionIdzoneId指定地域及可用区可以避免挂载失败。
  2. 执行以下命令创建Windows环境下的StorageClass。
    kubectl create -f storageclass.yaml

步骤二:为Windows节点安装Flexvolume插件

  1. 使用以下YAML示例创建flexvolume-windows.yaml文件。
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      labels:
        app: flexvolume-windows
      name: flexvolume-windows
      namespace: kube-system
    spec:
      selector:
        matchLabels:
          k8s-volume: flexvolume
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          metadata:
          labels:
            k8s-volume: flexvolume
          annotations:
            scheduler.alpha.kubernetes.io/critical-pod: ""
        spec:
          tolerations:
          - operator: Exists
          /*since 1.18, we can specify "hostNetwork: true" for Windows workloads, so we can deploy an application without NetworkReady.*/
          hostNetwork: true
          priorityClassName: system-node-critical
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                    - windows
                - matchExpressions:
                  - key: type
                    operator: NotIn
                    values:
                    - virtual-kubelet
                  - key: kubernetes.io/os
                    operator: In
                    values:
                    - windows
          containers:
          - name: acs-flexvolume
            command:
            - pwsh.exe
            - -NoLogo
            - -NonInteractive
            - -File
            - entrypoint.ps1
            /*根据不同集群的地域,您需修改以下镜像地址中的地域cn-hangzhou信息。*/
            image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/flexvolume-windows:v1.0.0
            imagePullPolicy: Always
            volumeMounts:
            - name: host-flexvolume-binary
              mountPath: c:/host/usr/libexec/kubernetes/kubelet-plugins/volume/exec
            - name: host-flexvolume-socket
              mountPath: c:/host/etc/kubernetes/flexvolume/socket
            - name: host-log
              mountPath: c:/var/log/alicloud
            /*use managed addon token from Alibaba Cloud*/
            - name: managed-addon-token
              mountPath: c:/var/addon/
              readOnly: true
         /* or specify the customized access key of Alibaba Cloud*/
         /* - name: customized-access-key*/
         /*   mountPath: c:/host/etc/.volumeak*/
         /* env:*/
         /* - name: ACCESS_KEY_ID*/
         /*   value: ""*/
         /* - name: ACCESS_KEY_SECRET*/
         /*   value: ""*/
          volumes:
          - name: host-flexvolume-binary
            hostPath:
              path: c:/usr/libexec/kubernetes/kubelet-plugins/volume/exec
              type: DirectoryOrCreate
          - name: host-flexvolume-socket
            hostPath:
              path: c:/etc/kubernetes/flexvolume/socket
              type: DirectoryOrCreate
          - name: host-log
            hostPath:
              path: c:/var/log/alicloud
              type: DirectoryOrCreate
          - name: managed-addon-token
            secret:
              defaultMode: 420
              optional: true
              items:
              - key: addon.token.config
                path: token-config
              secretName: addon.csi.token
         /* - name: customized-access-key*/
         /*   hostPath:*/
         /*     path: c:/etc/.volumeak*/
         /*     type: DirectoryOrCreate*/
       
  2. 执行以下命令为所有的Windows工作节点安装Flexvolume插件。
    kubectl create -f flexvolume-windows.yaml
  3. DaemonSet部署成功后,执行以下命令可以在每个WIndows节点上查看FlexVolume插件alicloud-disk-controller。
    ls C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\alicloud~disk.exe\
    预期输出:
    Directory: C:\usr\libexec\kubernetes\kubelet-plugins\volume\exec\alicloud~disk.exe
    
    Mode                LastWriteTime         Length Name
    ----                -------------         ------ ----
    -a----        8/12/2021   2:45 PM        5636096 disk.exe

步骤三:创建云盘并挂载到Windows容器中

以有状态工作负载(StatefulSet)为例,您可以在volumeClaimTemplates中定义需要的存储类(StroageClass),Windows工作节点上的FlexVolume插件会自动完成所有操作:验证您的账户信息,创建阿里云云盘,并挂载到阿里云服务器上,最后挂载到Windows容器中。

apiVersion: v1
kind: Service
metadata:
  labels:
    app: web-windows-disk
  name: web-windows-disk
spec:
  type: LoadBalancer
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: web-windows-disk
---
apiVersion: v1
kind: Service
metadata:
  name: web-windows-disk-headless
  labels:
    app: web-windows-disk
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  clusterIP: None
  selector:
    app: web-windows-disk
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web-windows-disk
spec:
  serviceName: "web-windows-disk-headless"
  selector:
    matchLabels:
      app: web-windows-disk
  replicas: 1
  template:
    metadata:
      labels:
        app: web-windows-disk
    spec:
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: os
        value: windows
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: In
                values:
                - windows
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - windows
      containers:
      - name: windows
        /*根据不同集群的地域,您需要修改以下镜像地址中的地域cn-hangzhou信息。*/
        image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/sample-web-windows:v1.0.1
        volumeMounts:
          - name: ssd
            mountPath: c:/openresty/logs
  volumeClaimTemplates:
  - metadata:
      name: ssd
    spec:
      accessModes: 
      - ReadWriteOnce
      /*请选择您集群所在的地域、可用区所支持的云盘类型。*/
      storageClassName: alicloud-disk-ssd-windows
      resources:
        requests:
          storage: 50Gi 
   

StatefulSet部署完成后,执行以下命令可查看到持久化存储卷(PersitentVolume)资源对象。
kubectl get pv
预期输出:
NAME                     STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS                AGE
ssd-web-windows-disk-0   Bound    d-wz9dihqdkusysc56****   50Gi       RWO            alicloud-disk-ssd-windows   5s

步骤四:验证云盘的持久化存储

在持久化存储卷(PersitentVolume)的支持下,部署的工作负载在被删除后,其访问日志仍然可以被保留。
  1. 执行以下命令访问部署的Windows应用web-windows-disk
    for i in {1..2}; do; curl $(kubectl get svc web-windows-disk --template '{{ (index .status.loadBalancer.ingress 0).ip }}'); done
  2. 执行以下命令查看web-windows-disk-0应用的日志。
    kubectl logs web-windows-disk-0
    预期输出:
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:28:28 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:28:32 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
  3. 执行以下命令删除web-windows-disk-0应用。
    kubectl delete pod web-windows-disk-0
  4. 执行以下命令访问重新部署的Windows应用web-windows-disk
    for i in {1..3}; do; curl $(kubectl get svc web-windows-disk --template '{{ (index .status.loadBalancer.ingress 0).ip }}'); done
  5. 执行以下命令查看web-windows-disk-0应用的日志。
    kubectl logs web-windows-disk-0
    预期输出:
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:28:28 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:28:32 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:56:40 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:56:40 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:56:41 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"

    从前后两次web-windows-disk-0应用日志的对比可得,部署的工作负载web-windows-disk在被删除后,其访问日志仍然被保留。

为Windows容器配置基于SMB的文件存储

阿里云为ECS提供阿里云文件存储NAS的分布式文件系统。阿里云文件存储NAS基于POSIX文件接口,天然适配原生操作系统,提供共享访问,同时保证数据一致性和锁互斥。

SMB(Server Message Block)是阿里云文件存储NAS实现的一种协议,在Windows平台上工作良好。和阿里云云盘FlexVolume插件不同的是,使用SMB Flexvolume插件并不依赖Linux工作节点或Kubernetes控制器。

没有Kubernetes控制器意味着SMB FlexVolume插件仅能处理静态的持久化存储卷(PersitentVolume)资源对象,而无法利用存储类(StorageClass)动态分配卷。

步骤一:为Windows节点安装Flexvolume插件

为所有的Windows工作节点安装Flexvolume插件。具体操作,请参见为Windows容器配置阿里云云盘的步骤二:为Windows节点安装Flexvolume插件。

步骤二:使用SMB创建持久卷并挂载到Windows容器中

  1. 创建持久化存储卷。
    使用以下YAML示例创建PV及PVC,向阿里云文件系统NAS申请10 GiB的文件存储。
    apiVersion: v1
    kind: PersistentVolume
    metadata:
      labels:
        alicloud-pvname: pv-smb-10g
      name: pv-smb-10g
    spec:
      persistentVolumeReclaimPolicy: Retain
      accessModes:
      - ReadWriteMany
      capacity:
        storage: 10Gi
      flexVolume:
        driver: alicloud/smb.exe
        options:
          server: 25f3f4819c-e****.cn-hangzhou.nas.aliyuncs.com
          path: /myshare/web-windows-logs
          user: workgroup/Administrator
          password: ******
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: web-windows-smb
    spec:
      selector:
        matchLabels:
          alicloud-pvname: pv-smb-10g
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
       
    配置项的部分参数说明如下所示:
    参数 说明
    driver 挂载存储卷的驱动类型,必须为alicloud/smb.exe
    server NAS文件系统中挂载点的挂载地址,必须与Windows节点的VPC在同一地域。
    path NAS在容器中挂载的位置,必须以/myshare/为父目录并创建子目录。例如,/myshare/web-windows-logs
    user 登录ECS的用户名,建议使用workgroup/administrator
    password 登录ECS的用户密码。
  2. 使用以下YAML示例创建Service及Deployment,并挂载已创建的持久化存储卷。
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: web-windows-smb
      name: web-windows-smb
    spec:
      type: LoadBalancer
      ports:
        - port: 80
          protocol: TCP
          targetPort: 80
      selector:
        app: web-windows-smb
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-windows-smb
    spec:
      selector:
        matchLabels:
          app: web-windows-smb
      replicas: 1
      template:
        metadata:
          labels:
            app: web-windows-smb
        spec:
          restartPolicy: Always
          terminationGracePeriodSeconds: 30
          tolerations:
          - key: os
            value: windows
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                    - windows
                - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                    - windows
          containers:
          - name: windows
            /*根据不同集群的地域,您需要修改以下镜像地址中的地域cn-hangzhou信息。*/
            image: registry-vpc.cn-hangzhou.aliyuncs.com/acs/sample-web-windows:v1.0.1
            volumeMounts:
              - name: disk-smb
                mountPath: c:/openresty/logs
          volumes:
          - name: disk-smb
            persistentVolumeClaim:
              claimName: web-windows-smb
       
  3. Deployment部署完成后,执行以下命令查看应用及持久化存储卷(PersitentVolume)资源对象。
    • 执行以下命令查看部署的工作负载Deployment。
      kubectl get pod
      预期输出:
      NAME                               READY   STATUS              RESTARTS   AGE
      web-windows-smb-584676df65-t4mmh   1/1     Running             0          26d
    • 执行以下命令查看持久化存储卷(PersitentVolume)资源对象。
      kubectl get pv
      预期输出:
      NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                       STORAGECLASS      REASON   AGE
      pv-smb-10g   10Gi       ReadWriteMany  Retain           Bound       default/web-windows-smb                                10d

步骤四:验证NAS的持久化存储

在持久化存储卷(PersitentVolume)的支持下,部署的工作负载在被删除后,其访问日志仍然可以被保留。
  1. 执行以下命令访问部署的Windows应用web-windows-smb
    for i in {1..2}; do; curl $(kubectl get svc web-windows-smb --template '{{ (index .status.loadBalancer.ingress 0).ip }}'); done
  2. 执行以下命令查看web-windows-smb-584676df65-t4mmh应用的日志。
    kubectl logs web-windows-smb-584676df65-t4mmh
    预期输出:
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:28:28 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:28:32 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
  3. 执行以下命令删除web-windows-smb-584676df65-t4mmh应用。
    kubectl delete pod web-windows-smb-584676df65-t4mmh
  4. 执行以下命令访问重新部署的Windows应用web-windows-smb
    for i in {1..3}; do; curl $(kubectl get svc web-windows-smb --template '{{ (index .status.loadBalancer.ingress 0).ip }}'); done
  5. 执行以下命令查看应用。
    kubectl get pod
    预期输出:
    NAME                               READY   STATUS              RESTARTS   AGE
    web-windows-smb-584676df65-87bpz   1/1     Running             0          26d
  6. 执行以下命令查看web-windows-smb-584676df65-87bpz应用的日志。
    kubectl logs web-windows-smb-584676df65-87bpz
    预期输出:
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:28:28 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:28:32 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    openresty started...
    x.x.x.x - - [14/Aug/2021:18:56:40 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:56:40 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"
    x.x.x.x - - [14/Aug/2021:18:56:41 +0800] "GET / HTTP/1.1" 200 674 "-" "curl/7.64.1"

    从前后两次web-windows-smb-****应用日志的对比可得,部署的工作负载web-windows-smb-584676df65-t4mmh在被删除后,其访问日志仍然被保留。