Container Service for Kubernetes:Accelerate OSS access from edge nodes with Fluid

Prerequisites

• An ACK Edge cluster of version 1.18 or later is created.

• An edge node pool is created and edge nodes are added.

• The cloud-native AI suite is installed with the ack-fluid add-on deployed.

  Important

  If open-source Fluid is installed, uninstall it before deploying ack-fluid.

  • If the cloud-native AI suite is not installed: enable Fluid data acceleration when you install the suite.

  • If the cloud-native AI suite is already installed: deploy ack-fluid on the Cloud-native AI Suite page of the ACK Console.

• The cluster is connected with kubectl.

• OSS is activated.

Step 1: Prepare OSS data

1. Download the test data to an ECS instance.

   wget https://archive.apache.org/dist/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz

2. Upload the downloaded test data to an OSS bucket.

   Important

   These steps use Alibaba Cloud Linux 3.2104 LTS 64-bit. For other systems, see ossutil quick start and ossutil command reference 1.0.

   1. Install ossutil.

   2. Create a bucket named examplebucket.

      • Create examplebucket.

        ossutil mb oss://examplebucket

      • examplebucket created successfully:

        0.668238(s) elapsed

   3. Upload the test data to examplebucket.

      ossutil cp spark-3.0.1-bin-hadoop2.7.tgz oss://examplebucket

Step 2: Create a Dataset and a JindoRuntime

1. Create a file named mySecret.yaml with the following content. Replace xxx with the AccessKey ID and AccessKey secret that have read access to the OSS bucket you created in Step 1.

   apiVersion: v1
   kind: Secret
   metadata:
     name: mysecret
   stringData:
     fs.oss.accessKeyId: xxx
     fs.oss.accessKeySecret: xxx

2. Apply the Secret. Kubernetes encrypts the stored values so they are not exposed as plaintext.

   kubectl create -f mySecret.yaml

3. Create a resource.yaml file with the following content:

   • Define a Dataset that describes the remote storage and underlying file system (UFS).

   • Create a JindoRuntime to launch a JindoFS cluster for data caching.

   apiVersion: data.fluid.io/v1alpha1
   kind: Dataset
   metadata:
     name: hadoop
   spec:
     nodeAffinity:
       required:
         nodeSelectorTerms:
           - matchExpressions:
             - key: alibabacloud.com/nodepool-id
               operator: In
               values:
                 - npxxxxxxxxxxxxxx
     mounts:
       - mountPoint: oss://<oss_bucket>/<bucket_dir>
         options:
           fs.oss.endpoint: <oss_endpoint>
         name: hadoop
         path: "/"
         encryptOptions:
           - name: fs.oss.accessKeyId
             valueFrom:
               secretKeyRef:
                 name: mysecret
                 key: fs.oss.accessKeyId
           - name: fs.oss.accessKeySecret
             valueFrom:
               secretKeyRef:
                 name: mysecret
                 key: fs.oss.accessKeySecret
   ---
   apiVersion: data.fluid.io/v1alpha1
   kind: JindoRuntime
   metadata:
     name: hadoop
   spec:
     nodeSelector:
       alibabacloud.com/nodepool-id: npxxxxxxxxxxxxxx
     replicas: 2
     tieredstore:
       levels:
         - mediumtype: MEM
           path: /dev/shm
           volumeType: emptyDir
           quota: 2Gi
           high: "0.99"
           low: "0.95"

   Note

   • In an ACK Edge cluster, use nodeAffinity and nodeSelector to deploy the Dataset and JindoRuntime to the same node pool so nodes within the pool can communicate.

   • Edge node management and OSS access both require cloud-to-edge communication. Maintain sufficient network bandwidth to avoid destabilizing the control channel.

   Parameters:

   Parameter	Description
   mountPoint	oss://<oss_bucket>/<bucket_dir>. UFS mount path. Must point to a directory. A single file cannot be mounted. Do not include the endpoint.
   fs.oss.endpoint	Public or internal endpoint of the OSS bucket. See Regions and endpoints.
   replicas	Number of workers in the JindoFS caching cluster.
   mediumtype	Cache storage medium. Valid values: HDD, SSD, MEM.
   path	Local storage path on the worker node. Only one path is allowed. Required when mediumtype is MEM to store data such as logs.
   quota	Maximum cache size per worker.
   high	Cache eviction threshold (high watermark). When usage exceeds this ratio, eviction begins.
   low	Cache retention threshold (low watermark). Eviction stops when usage drops to this ratio.

4. Create the Dataset and JindoRuntime.

   kubectl create -f resource.yaml

5. Verify that the Dataset is bound.

   kubectl get dataset hadoop

   Expected output:

   NAME     UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
   hadoop        210MiB       0.00B    4.00GiB              0.0%          Bound   1h

6. Verify that the JindoRuntime is ready.

   kubectl get jindoruntime hadoop

   Expected output:

   NAME     MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
   hadoop   Ready          Ready          Ready        4m45s

7. Verify that the persistent volume (PV) and persistent volume claim (PVC) are created.

   kubectl get pv,pvc

   Expected output:

   NAME                      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM            STORAGECLASS   REASON   AGE
   persistentvolume/hadoop   100Gi      RWX            Retain           Bound    default/hadoop                           52m
   
   NAME                           STATUS   VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   persistentvolumeclaim/hadoop   Bound    hadoop   100Gi      RWX                           52m

The Dataset and JindoRuntime are ready when all phases show Ready and the PVC status is Bound.

Step 3: Test data access acceleration

Deploy a pod that mounts the Dataset PVC and compare file read times before and after data is cached.

1. Create an app.yaml file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: demo-app
   spec:
     nodeSelector:
       alibabacloud.com/nodepool-id: npxxxxxxxxxxxxx
     containers:
       - name: demo
         image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
         volumeMounts:
           - mountPath: /data
             name: hadoop
     volumes:
       - name: hadoop
         persistentVolumeClaim:
           claimName: hadoop

   Note

   In an ACK Edge cluster, use nodeSelector to deploy the test pod to the node pool in Step 2.

2. Deploy the pod.

   kubectl create -f app.yaml

3. Open a shell in the pod and check the file size.

   kubectl exec -it demo-app -- bash
   du -sh /data/spark-3.0.1-bin-hadoop2.7.tgz

   Expected output:

   210M    /data/spark-3.0.1-bin-hadoop2.7.tgz

4. Measure the initial read time. This access comes directly from OSS, with no cache.

   time cp /data/spark-3.0.1-bin-hadoop2.7.tgz /dev/null

   Expected output:

   real    0m18.386s
   user    0m0.002s
   sys    0m0.105s

   Reading the file takes about 18 seconds.

5. Check the cached data after the read.

   kubectl get dataset hadoop

   Expected output:

   NAME     UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
   hadoop   210.00MiB       210.00MiB    4.00GiB        100.0%           Bound   1h

   The full 210 MiB is now cached in local storage.

6. Delete and recreate the pod to clear the OS page cache, so the next read comes from JindoFS cache rather than memory.

   kubectl delete -f app.yaml && kubectl create -f app.yaml

7. Measure the read time again with data served from cache.

   kubectl exec -it demo-app -- bash
   time cp /data/spark-3.0.1-bin-hadoop2.7.tgz /dev/null

   Expected output:

   real    0m0.048s
   user    0m0.001s
   sys     0m0.046s

   With JindoFS cache, the same file read completes in 48 milliseconds — more than 300 times faster than the direct OSS access.

(Optional) Clean up

When data acceleration is no longer needed, delete the pod, the Dataset, and the JindoRuntime.

Delete the pod:

kubectl delete pod demo-app

Delete the Dataset and JindoRuntime:

kubectl delete dataset hadoop