Container Service for Kubernetes:Accelerate Spark Jobs with Fluid

Prerequisites

Before you begin, ensure that you have:

• The ack-spark-operator component installed.

• The cloud-native AI suite deployed with the ack-fluid component enabled.

• Test data uploaded to an OSS bucket.

How it works

Fluid routes Spark data reads through JindoRuntime worker pods. On first access, JindoRuntime fetches data from OSS and writes it to local SATA HDDs. Subsequent reads hit the local cache, eliminating network round-trips to OSS.

Two access methods are available:

Step 1: Create a dedicated node pool for Fluid

Create a node pool named fluid for JindoRuntime worker pods. This example uses three ecs.d1ne.4xlarge instances (network-enhanced big data family), each with eight 5,905 GB local SATA HDDs mounted at /mnt/disk1 through /mnt/disk8.

Add this label and taint to each node:

• Label: fluid-cloudnative.github.io/node="true"

• Taint: fluid-cloudnative.github.io/node="true":NoSchedule

See Create and manage a node pool for setup and Best practices for the cache optimization policies of Fluid for instance types.

Step 2: Create a dataset

A Dataset is a Fluid custom resource that maps an OSS path to a cache-backed volume, so Spark pods read from the cache instead of OSS directly.

1. Create fluid-oss-secret.yaml. Replace <ACCESS_KEY_ID> and <ACCESS_KEY_SECRET> with your AccessKey pair.

   apiVersion: v1
   kind: Secret
   metadata:
     name: fluid-oss-secret
     namespace: spark
   stringData:
     OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
     OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET>

2. Create the Secret:

   kubectl create -f fluid-oss-secret.yaml

   Expected output:

   secret/fluid-oss-secret created

3. Create spark-fluid-dataset.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: Dataset
   metadata:
     name: spark
     namespace: spark
   spec:
     mounts:
     - name: spark
       # Required. The OSS path to cache. Replace <OSS_BUCKET> with your bucket name.
       mountPoint: oss://<OSS_BUCKET>/
       path: /
       options:
         # Required. The OSS endpoint. Example for China (Beijing) internal: oss-cn-beijing-internal.aliyuncs.com
         fs.oss.endpoint: <OSS_ENDPOINT>
       encryptOptions:
       - name: fs.oss.accessKeyId
         valueFrom:
           secretKeyRef:
             name: fluid-oss-secret
             key: OSS_ACCESS_KEY_ID
       - name: fs.oss.accessKeySecret
         valueFrom:
           secretKeyRef:
             name: fluid-oss-secret
             key: OSS_ACCESS_KEY_SECRET
     # Data is cached only on nodes matching this affinity rule.
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: fluid-cloudnative.github.io/node
             operator: In
             values:
             - "true"
     # Required for nodes with the fluid taint.
     tolerations:
     - key: fluid-cloudnative.github.io/node
       operator: Equal
       value: "true"
       effect: NoSchedule

   Key parameters:

   Parameter	Description
   mountPoint	The OSS path to cache.
   fs.oss.endpoint	The OSS bucket endpoint. For the China (Beijing) region internal endpoint, use oss-cn-beijing-internal.aliyuncs.com.
   encryptOptions	Reads OSS credentials from the fluid-oss-secret Secret.

4. Create the Dataset:

   kubectl create -f spark-fluid-dataset.yaml

   Expected output:

   dataset.data.fluid.io/spark created

5. Verify the Dataset status:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE      HCFS URL   TOTAL FILES   CACHE HIT RATIO   AGE
   spark                                                                  NotBound                                              58m

   NotBound is expected — the Dataset binds to a runtime in the next step.

Step 3: Create a JindoRuntime

A JindoRuntime deploys JindoFS master and worker pods that back the Dataset with local disk cache.

1. Create spark-fluid-jindoruntime.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: JindoRuntime
   metadata:
     # Required. Must match the Dataset name.
     name: spark
     namespace: spark
   spec:
     # Required. Number of worker pods — one per cache node.
     replicas: 3
     tieredstore:
       levels:
       - mediumtype: HDD
         volumeType: hostPath
         # Required. Set based on the number of disks on each node.
         path: /mnt/disk1,/mnt/disk2,/mnt/disk3,/mnt/disk4,/mnt/disk5,/mnt/disk6,/mnt/disk7,/mnt/disk8
         # Required. Cache quota per disk. Total cache = 8 x 5500Gi = 43.95 TiB per worker.
         quotaList: 5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi
         # Optional. Start evicting cache when disk usage reaches 99%.
         high: "0.99"
         # Optional. Stop evicting when disk usage drops to 95%.
         low: "0.95"
     worker:
       resources:
         requests:
           cpu: 14
           memory: 56Gi
         limits:
           cpu: 14
           memory: 56Gi

   Key parameters:

   Parameter	Description
   replicas	Number of JindoRuntime worker pods.
   mediumtype	Cache storage type. HDD uses local hard disks.
   path	Mount paths of the local disks on each worker node.
   quotaList	Maximum cache size per disk.
   high	Disk usage ratio that triggers cache eviction. "0.99" starts eviction at 99% disk usage.
   low	Disk usage ratio at which cache eviction stops. "0.95" stops eviction at 95% usage.

2. Create the JindoRuntime:

   kubectl create -f spark-fluid-jindoruntime.yaml

   Expected output:

   jindoruntime.data.fluid.io/spark created

3. Verify the JindoRuntime is ready:

   kubectl get -n spark jindoruntime spark

   Expected output:

   NAME    MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
   spark   Ready          Ready          Ready        2m28s

   All three phases must show Ready before proceeding.

4. Verify the Dataset is now bound:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   HCFS URL                             TOTAL FILES     CACHE HIT RATIO   AGE
   spark   [Calculating]    0.00B    128.91TiB                            Bound   spark-jindofs-master-0.spark:19434   [Calculating]                     2m5s

   Bound confirms the Dataset is ready to serve cached data.

Step 4: (Optional) Prefetch data

By default, the first Spark job fetches data from OSS at network speed. Use a DataLoad to preload data into the cache so even the first run reads locally.

1. Create spark-fluid-dataload.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: DataLoad
   metadata:
     name: spark
     namespace: spark
   spec:
     dataset:
       name: spark
       namespace: spark
     loadMetadata: true

2. Create the DataLoad:

   kubectl create -f spark-fluid-dataload.yaml

   Expected output:

   dataload.data.fluid.io/spark created

3. Monitor prefetching progress:

   kubectl get -n spark dataload spark -w

   Expected output:

   NAME    DATASET   PHASE       AGE     DURATION
   spark   spark     Executing   20s     Unfinished
   spark   spark     Complete    9m31s   8m37s

   Prefetching completes when the phase shows Complete.

4. Verify cached data volume:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED      CACHE CAPACITY   CACHED PERCENTAGE   PHASE   HCFS URL                             TOTAL FILES     CACHE HIT RATIO   AGE
   spark   0.00B            326.85GiB   128.91TiB        0.0%                Bound   spark-jindofs-master-0.spark:19434   [Calculating]                     19m

   CACHED shows 326.85GiB, confirming data is preloaded.

Step 5: Run a Spark job

Choose a method based on your setup:

• Use POSIX if your Spark job reads local files. Fluid mounts a PVC accessible via file:// paths — no code changes needed.

• Use HCFS if your job already uses oss://, s3://, or s3a:// paths, or you want to route OSS access through the cache. Requires a custom Spark image with JindoSDK.

Step 6: (Optional) Clean up

Delete resources in this order to avoid dependency conflicts:

kubectl delete -f spark-pagerank-fluid-posix.yaml
kubectl delete -f spark-pagerank-fluid-hcfs.yaml
kubectl delete -f spark-fluid-dataload.yaml
kubectl delete -f spark-fluid-jindoruntime.yaml
kubectl delete -f spark-fluid-dataset.yaml
kubectl delete -f fluid-oss-secret.yaml

Next steps

• Elastic datasets — Fluid's dataset abstraction and supported runtimes.

• Best practices for the cache optimization policies of Fluid — Optimize cache performance for your workloads.