Container Service for Kubernetes:Accelerate Spark Jobs with Fluid

Prerequisites

Before you begin, ensure that you have:

• The ack-spark-operator component installed. For more information, see Step 1: Install the ack-spark-operator component.

• The cloud-native AI suite deployed with the ack-fluid component enabled. For more information, see Install the cloud-native AI suite.

• Test data uploaded to an OSS bucket. For more information, see Prepare and upload test data to an OSS bucket.

How it works

Fluid intercepts data access from Spark jobs and routes reads through JindoRuntime worker pods. On the first access, JindoRuntime fetches data from OSS and writes it to local SATA HDDs. Subsequent reads hit the local cache, eliminating repeated network round-trips to OSS.

Two access methods are available:

Step 1: Create a dedicated node pool for Fluid

Create a node pool named fluid for the JindoRuntime worker pods. This example uses three ecs.d1ne.4xlarge instances (network-enhanced big data family). Each node has eight 5,905 GB high-throughput local SATA HDDs, formatted and mounted at /mnt/disk1 through /mnt/disk8.

Add the following label and taint to each node:

• Label: fluid-cloudnative.github.io/node="true"

• Taint: fluid-cloudnative.github.io/node="true":NoSchedule

For guidance on creating a node pool, see Create and manage a node pool. For instance type recommendations, see Best practices for the cache optimization policies of Fluid.

Step 2: Create a dataset

A Dataset is a Fluid custom resource that maps an OSS path to a cache-backed volume. Spark pods access data through this volume instead of going directly to OSS.

1. Create fluid-oss-secret.yaml to store your OSS credentials. Replace <ACCESS_KEY_ID> and <ACCESS_KEY_SECRET> with your Alibaba Cloud AccessKey pair.

   apiVersion: v1
   kind: Secret
   metadata:
     name: fluid-oss-secret
     namespace: spark
   stringData:
     OSS_ACCESS_KEY_ID: <ACCESS_KEY_ID>
     OSS_ACCESS_KEY_SECRET: <ACCESS_KEY_SECRET>

2. Create the Secret:

   kubectl create -f fluid-oss-secret.yaml

   Expected output:

   secret/fluid-oss-secret created

3. Create spark-fluid-dataset.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: Dataset
   metadata:
     name: spark
     namespace: spark
   spec:
     mounts:
     - name: spark
       # Required. The OSS path to cache. Replace <OSS_BUCKET> with your bucket name.
       mountPoint: oss://<OSS_BUCKET>/
       path: /
       options:
         # Required. The OSS endpoint. Example for China (Beijing) internal: oss-cn-beijing-internal.aliyuncs.com
         fs.oss.endpoint: <OSS_ENDPOINT>
       encryptOptions:
       - name: fs.oss.accessKeyId
         valueFrom:
           secretKeyRef:
             name: fluid-oss-secret
             key: OSS_ACCESS_KEY_ID
       - name: fs.oss.accessKeySecret
         valueFrom:
           secretKeyRef:
             name: fluid-oss-secret
             key: OSS_ACCESS_KEY_SECRET
     # Data is cached only on nodes matching this affinity rule.
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: fluid-cloudnative.github.io/node
             operator: In
             values:
             - "true"
     # Required for nodes with the fluid taint.
     tolerations:
     - key: fluid-cloudnative.github.io/node
       operator: Equal
       value: "true"
       effect: NoSchedule

   Key parameters:

   Parameter	Description
   mountPoint	The OSS path to cache.
   fs.oss.endpoint	The OSS bucket endpoint. For the China (Beijing) region internal endpoint, use oss-cn-beijing-internal.aliyuncs.com.
   encryptOptions	Reads OSS credentials from the fluid-oss-secret Secret.

4. Create the Dataset:

   kubectl create -f spark-fluid-dataset.yaml

   Expected output:

   dataset.data.fluid.io/spark created

5. Verify the Dataset status:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE      HCFS URL   TOTAL FILES   CACHE HIT RATIO   AGE
   spark                                                                  NotBound                                              58m

   The NotBound phase is expected. The Dataset binds to a runtime in the next step.

Step 3: Create a JindoRuntime

A JindoRuntime deploys the JindoFS master and worker pods that back the Dataset with local disk cache.

1. Create spark-fluid-jindoruntime.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: JindoRuntime
   metadata:
     # Required. Must match the Dataset name.
     name: spark
     namespace: spark
   spec:
     # Required. Number of worker pods — one per cache node.
     replicas: 3
     tieredstore:
       levels:
       - mediumtype: HDD
         volumeType: hostPath
         # Required. Set based on the number of disks on each node.
         path: /mnt/disk1,/mnt/disk2,/mnt/disk3,/mnt/disk4,/mnt/disk5,/mnt/disk6,/mnt/disk7,/mnt/disk8
         # Required. Cache quota per disk. Total cache = 8 x 5500Gi = 43.95 TiB per worker.
         quotaList: 5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi,5500Gi
         # Optional. Start evicting cache when disk usage reaches 99%.
         high: "0.99"
         # Optional. Stop evicting when disk usage drops to 95%.
         low: "0.95"
     worker:
       resources:
         requests:
           cpu: 14
           memory: 56Gi
         limits:
           cpu: 14
           memory: 56Gi

   Key parameters:

   Parameter	Description
   replicas	Number of JindoRuntime worker pods.
   mediumtype	Cache storage type. HDD uses local hard disks.
   path	Mount paths of the local disks on each worker node.
   quotaList	Maximum cache size per disk. Each value maps to the corresponding path.
   high	Disk usage ratio that triggers cache eviction. For example, "0.99" means eviction starts at 99% disk usage.
   low	Disk usage ratio at which cache eviction stops. For example, "0.95" means eviction stops when disk usage drops to 95%.

2. Create the JindoRuntime:

   kubectl create -f spark-fluid-jindoruntime.yaml

   Expected output:

   jindoruntime.data.fluid.io/spark created

3. Wait for the JindoRuntime to be ready:

   kubectl get -n spark jindoruntime spark

   Expected output:

   NAME    MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
   spark   Ready          Ready          Ready        2m28s

   All three phases must show Ready before proceeding.

4. Verify the Dataset is now bound:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   HCFS URL                             TOTAL FILES     CACHE HIT RATIO   AGE
   spark   [Calculating]    0.00B    128.91TiB                            Bound   spark-jindofs-master-0.spark:19434   [Calculating]                     2m5s

   The Bound phase confirms the Dataset is ready to serve cached data.

Step 4: (Optional) Prefetch data

The cache is empty until Spark jobs run and access data. The first job run fetches data from OSS at regular network speeds. To warm the cache before any job runs — so even the first job benefits from local reads — use a DataLoad to preload OSS data into the cache.

1. Create spark-fluid-dataload.yaml:

   apiVersion: data.fluid.io/v1alpha1
   kind: DataLoad
   metadata:
     name: spark
     namespace: spark
   spec:
     dataset:
       name: spark
       namespace: spark
     loadMetadata: true

2. Create the DataLoad:

   kubectl create -f spark-fluid-dataload.yaml

   Expected output:

   dataload.data.fluid.io/spark created

3. Monitor prefetching progress:

   kubectl get -n spark dataload spark -w

   Expected output:

   NAME    DATASET   PHASE       AGE     DURATION
   spark   spark     Executing   20s     Unfinished
   spark   spark     Complete    9m31s   8m37s

   Prefetching is complete when the phase shows Complete.

4. Verify cached data volume:

   kubectl get -n spark dataset spark -o wide

   Expected output:

   NAME    UFS TOTAL SIZE   CACHED      CACHE CAPACITY   CACHED PERCENTAGE   PHASE   HCFS URL                             TOTAL FILES     CACHE HIT RATIO   AGE
   spark   0.00B            326.85GiB   128.91TiB        0.0%                Bound   spark-jindofs-master-0.spark:19434   [Calculating]                     19m

   The CACHED column shows 326.85GiB, confirming that data is preloaded.

Step 5: Run a Spark job

Choose a method based on your setup:

• Use POSIX if your Spark job reads local files and you want no code changes. Fluid mounts a PVC that Spark accesses via file:// paths.

• Use HCFS if your job already uses oss://, s3://, or s3a:// paths, or you want to route existing OSS access through the cache. This method requires a custom Spark image with JindoSDK.

Step 6: (Optional) Clean up

Delete resources in the following order to avoid dependency conflicts:

kubectl delete -f spark-pagerank-fluid-posix.yaml
kubectl delete -f spark-pagerank-fluid-hcfs.yaml
kubectl delete -f spark-fluid-dataload.yaml
kubectl delete -f spark-fluid-jindoruntime.yaml
kubectl delete -f spark-fluid-dataset.yaml
kubectl delete -f fluid-oss-secret.yaml

What's next

• Elastic datasets — Learn more about Fluid's dataset abstraction and supported runtimes.

• Best practices for the cache optimization policies of Fluid — Optimize cache performance for your workloads.