Manage the lifecycle of OSS buckets - Container Service for Kubernetes

Prerequisites

Ensure you have:

csi-plugin and csi-provisioner at version 1.24.2-5b34494d-aliyun or later (upgrade)
storage-operator at version 1.24.95-e2d0756-aliyun or later (upgrade)

Limitations

The only supported reclaim policy is Retain. Deleting a CNFS CRD preserves the bucket.
Archive and Cold Archive objects must be restored before read or write.
If versioning is enabled, retention policies and OSS-HDFS cannot be configured. To use them, set enableVersioning to None first.
After versioning is set to enabled, it cannot be disabled, only suspended.
The redundancyType and enableVersioning parameters require storage-operator v1.26.2-1de13b6-aliyun or later.
If redundancyType is ZRS, Cold Archive and Deep Cold Archive are not supported.

Method 1: Create a new OSS bucket with CNFS

Creates an OSS bucket named cnfs-oss-<clusterid> and mounts it to a Deployment and StatefulSet as a dynamically provisioned volume.

If a bucket with the specified name already exists, CNFS associates it with the ContainerNetworkFileSystem object instead of creating a new one.

Create the CNFS CRD and StorageClass

Create the Secret, ContainerNetworkFileSystem CRD, and StorageClass. Replace <clusterid> with your cluster ID.

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
stringData:
  akId: "xxxx"      # AccessKey ID for mounting the OSS bucket
  akSecret: "xxxx"  # AccessKey Secret for mounting the OSS bucket
---
apiVersion: storage.alibabacloud.com/v1beta1
kind: ContainerNetworkFileSystem
metadata:
  name: cnfs-oss-<clusterid>   # Set the CNFS CRD name to match the bucket name
spec:
  description: "cnfs-oss"
  type: oss
  reclaimPolicy: Retain         # Only Retain is supported. Deleting the CRD keeps the bucket.
  parameters:
    bucketName: cnfs-oss-<clusterid>   # Replace <clusterid> with your cluster ID; bucket name must be unique
    encryptType: "AES256"              # AES-256 server-side encryption
    storageType: "Standard"            # Storage class
    aclType: "private"                 # Bucket owner and authorized users only
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alibabacloud-cnfs-oss
parameters:
  containerNetworkFileSystem: cnfs-oss-<clusterid>           # Reference to the CNFS CRD
  otherOpts: -o max_stat_cache_size=0 -o allow_other         # Cache settings
  path: /
  # volumeAs: subpath   # Uncomment to auto-create a subpath for each PV
  csi.storage.k8s.io/node-publish-secret-name: oss-secret
  csi.storage.k8s.io/node-publish-secret-namespace: default
provisioner: ossplugin.csi.alibabacloud.com
reclaimPolicy: Retain
EOF

CNFS CRD parameters:

Parameter	Description	Required	Default
`description`	Description of the CNFS file system.	No	—
`type`	Volume type. Set to `oss`.	Yes	—
`reclaimPolicy`	Reclaim policy. Only `Retain` is supported.	Yes	—
`parameters.bucketName`	OSS bucket name. If a matching bucket exists, it is associated with the CNFS CRD; otherwise, a new bucket is created.	Yes	—
`parameters.storageType`	Storage class. Valid values: `Standard`, `IA` (Infrequent Access), `Archive`, `ColdArchive`. Archive and Cold Archive objects must be restored before read/write.	No	`Standard`
`parameters.redundancyType`	Storage redundancy type. `LRS`: single-zone; `ZRS` (zone-redundant): cross-zone. Requires storage-operator v1.26.2-1de13b6-aliyun or later. ZRS does not support Cold Archive or Deep Cold Archive.	No	`ZRS`
`parameters.encryptType`	Server-side encryption algorithm. Valid values: `None`, `AES256`, `SM4`.	No	—
`parameters.aclType`	ACL type. Valid values: `private` (owner and authorized users only), `public-read` (public read; owner and authorized users write), `public-read-write` (all users including anonymous can read and write — use with caution).	No	`private`
`parameters.enableVersioning`	Versioning status. Valid values: `enabled` , `suspended` , `None` . Once enabled, cannot be disabled — only suspended. Prevents retention policy or OSS-HDFS configuration. Requires storage-operator v1.26.2-1de13b6-aliyun or later. See Lifecycle billing.	No	`Enabled`

Create a PVC and sample workloads

Create a persistent volume claim (PVC), a Deployment, and a StatefulSet to verify the bucket mounts correctly.

cat << EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cnfs-oss-pvc
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: alibabacloud-cnfs-oss
  resources:
    requests:
      storage: 100Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cnfs-oss-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: "/data"
          name: cnfs-oss-pvc
      volumes:
      - name: cnfs-oss-pvc
        persistentVolumeClaim:
          claimName: cnfs-oss-pvc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cnfs-oss-sts
  labels:
    app: nginx
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: "/data"
          name: www
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadOnlyMany" ]
      storageClassName: "alibabacloud-cnfs-oss"
      resources:
        requests:
          storage: 100Gi
EOF

Verify the bucket and workloads

Confirm the OSS bucket and pods are ready.

Verify the CNFS CRD:

kubectl get cnfs/cnfs-oss-<clusterid> -o yaml

Expected output:

apiVersion: storage.alibabacloud.com/v1beta1
kind: ContainerNetworkFileSystem
...
status:
  conditions:
  - lastProbeTime: "2022-09-18 15:02:39"
    reason: The oss bucket is complete initialization.
    status: Ready
  fsAttributes:
    accessGroupName: DEFAULT_VPC_GROUP_NAME
    aclType: private
    bucketName: cnfs-oss-****
    encryptType: AES256
    endPoint:
      extranet: oss-****.aliyuncs.com
      internal: oss-****-internal.aliyuncs.com
    regionId: ****
    storageType: Standard
  status: Available

status.status shows Available when the bucket is ready. Status fields:

Field	Description
`status`	CNFS CRD status. Valid values: `Pending, Creating, Initialization (creating the file system), Available, Unavailable (recoverable), Fatal (not recoverable), Terminating`.
`conditions.lastProbeTime`	Last probe timestamp.
`conditions.reason`	Reason for the current status.
`conditions.status`	Readiness: `Ready` (available) or `NotReady`.
`fsAttributes.accessGroupName`	Permission group for the mount point. Currently, only DEFAULT_VPC_GROUP_NAME (the default VPC group) is supported.
`fsAttributes.encryptType`	Encryption algorithm: `None`, `AES256`, or `SM4`.
`fsAttributes.regionId`	ACK cluster region.
`fsAttributes.storageType`	Bucket storage class: `Standard`, `IA`, `Archive`, or `ColdArchive`.
`fsAttributes.redundancyType`	Storage redundancy type: `LRS` (locally redundant storage) or `ZRS` (zone-redundant storage).
`fsAttributes.aclType`	Bucket ACL type.
`fsAttributes.endPoint`	Endpoints: `extranet` (public) and `internal` (internal network).
`fsAttributes.enableVersioning`	Versioning status: `enabled`, `suspended`, or `None`.

Verify the pods:

kubectl get pod

Expected output:

NAME                                   READY   STATUS    RESTARTS   AGE
cnfs-oss-deployment-5864fd8d98-4****   1/1     Running   0          2m21s
cnfs-oss-sts-0                         1/1     Running   0          2m21s
cnfs-oss-sts-1                         1/1     Running   0          2m16s

All pods show Running, confirming the Deployment and StatefulSet mounted the OSS bucket.

Method 2: Use an existing OSS bucket

Creates a CNFS CRD that references an existing OSS bucket by name.

Create the CNFS CRD

cat <<EOF | kubectl apply -f -
apiVersion: storage.alibabacloud.com/v1beta1
kind: ContainerNetworkFileSystem
metadata:
  name: cnfs-oss-exist-bucket-name
spec:
  description: "cnfs-oss"
  type: oss
  reclaimPolicy: Retain
  parameters:
    bucketName: bucket-name   # Name of your existing OSS bucket
EOF

Verify the CNFS CRD

kubectl get cnfs/cnfs-oss-exist-bucket-name -o yaml

Expected output:

apiVersion: storage.alibabacloud.com/v1beta1
kind: ContainerNetworkFileSystem
...
status:
  conditions:
  - lastProbeTime: "2022-09-14 17:00:21"
    reason: The oss bucket is complete initialization.
    status: Ready
  fsAttributes:
    accessGroupName: DEFAULT_VPC_GROUP_NAME
    aclType: private
    bucketName: exist-bucket-name
    encryptType: AES256
    endPoint:
      extranet: oss-****.aliyuncs.com
      internal: oss-****-internal.aliyuncs.com
    regionId: ****
    storageType: Standard
  status: Available

Mount the bucket to a workload

Mount the bucket to your Deployment as a static or dynamic volume by following Create a PVC and sample workloads (Method 1), using this method's CNFS CRD.

Next steps

Configure lifecycle rules to transition or expire objects.
Review billing items for different storage classes and redundancy types.