All Products
Search

This Product

    Container Service for Kubernetes:Use Celeborn as an RSS for Spark jobs

    Document Center

    Container Service for Kubernetes:Use Celeborn as an RSS for Spark jobs

    Last Updated:Jun 15, 2026

    Celeborn manages intermediate shuffle data and spill data as a remote shuffle service (RSS). Deploy it in a Container Service for Kubernetes (ACK) cluster to use the RSS for Spark jobs.

    Benefits

    For big data frameworks such as MapReduce, Spark, and Flink, Celeborn as an RSS provides:

    • Push-based shuffle write: Mapper nodes skip local disk storage, ideal for cloud-native architectures with storage and compute separation.

    • Merge-based shuffle read: Data merges on worker nodes instead of reducers, avoiding random small-file I/O and network overhead from small transfers.

    • high availability: Celeborn master nodes use the Raft protocol for high availability and stability.

    • fault tolerance: dual replicas significantly reduce fetch failure probability.

    Prerequisites

    Cluster environment

    This example uses the following ACK cluster configuration:

    • The master process is deployed to the celeborn-master node pool with the following configuration:

      • node pool name: celeborn-master

      • Number of nodes: 3

      • ECS instance type: g8i.2xlarge

      • label: celeborn.apache.org/role=master

      • taint: celeborn.apache.org/role=master:NoSchedule

      • Data storage per node: /mnt/celeborn_ratis (1024 GB)

    • The worker process is deployed to the celeborn-worker node pool with the following configuration:

      • node pool name: celeborn-worker

      • Number of nodes: 5

      • ECS instance type: g8i.4xlarge

      • label: celeborn.apache.org/role=worker

      • taint: celeborn.apache.org/role=worker:NoSchedule

      • Data storage per node:

        • /mnt/disk1 (1024 GB)

        • /mnt/disk2 (1024 GB)

        • /mnt/disk3 (1024 GB)

        • /mnt/disk4 (1024 GB)

    Procedure overview

    Complete these steps to deploy Celeborn in an ACK cluster and run a sample Spark job.

    1. Build a Celeborn container image

      Download a Celeborn release, build a container image, and push it to your image repository to deploy the ack-celeborn component.

    2. Deploy the ack-celeborn component

      Use the ack-celeborn Helm chart from the ACK Marketplace to deploy a Celeborn cluster using the container image you built.

    3. Build a Spark container image

      Build a Spark container image with Celeborn and OSS dependencies, and push it to your image repository.

    4. Prepare and upload test data to OSS

      Generate a test dataset for the PageRank job and upload it to OSS.

    5. Run a sample Spark job

      Run a sample PageRank job and configure it to use Celeborn as the RSS.

    6. (Optional) Clean up resources

      After the tutorial, clean up the Spark job and other unneeded resources to avoid fees.

    Step 1: Build a Celeborn container image

    Download the required release (for example, version 0.5.2) from the official Celeborn website. Replace <IMAGE-REGISTRY> and <IMAGE-REPOSITORY> with your image registry and image name. You can modify PLATFORMS to set the target architecture. See Deploy Celeborn on Kubernetes. The docker buildx command requires Docker 19.03 or later. See Install and use Docker and Docker Compose.

    CELEBORN_VERSION=0.5.2               # The Celeborn version.

IMAGE_REGISTRY=<IMAGE-REGISTRY>      # The image registry, for example, docker.io.

IMAGE_REPOSITORY=<IMAGE-REPOSITORY>  # The image name, for example, apache/celeborn.

IMAGE_TAG=${CELEBORN_VERSION}        # The image tag. This example uses the Celeborn version as the tag.

PLATFORMS=linux/amd64                # The image platform architecture. To support multiple platforms, separate them with commas, for example, linux/amd64,linux/arm64.

# Download the release package.
wget https://downloads.apache.org/celeborn/celeborn-${CELEBORN_VERSION}/apache-celeborn-${CELEBORN_VERSION}-bin.tgz

# Extract the package.
tar -zxvf apache-celeborn-${CELEBORN_VERSION}-bin.tgz

# Go to the working directory.
cd apache-celeborn-${CELEBORN_VERSION}-bin

# Use Docker Buildx to build the image and push it to the image repository.
docker buildx build \
    --output=type=registry \
    --push \
    --platform=${PLATFORMS} \
    --tag=${IMAGE_REGISTRY}/${IMAGE_REPOSITORY}:${IMAGE_TAG} \
    -f docker/Dockerfile \
    .

    Step 2: Deploy the ack-celeborn component

    1. Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.

    2. On the Marketplace page, click the App Catalog tab, select ack-celeborn, and on the ack-celeborn page, click Deploy.

    3. In the Create panel, select a cluster and namespace, and then click Next.

    4. On the Parameters page, configure the parameters and then click OK.

      image:                         # Replace this with the address of the Celeborn image that you built in Step 1.
  registry: docker.io          # The image registry.
  repository: apache/celeborn  # The image name.
  tag: 0.5.2                   # The image tag.

celeborn:
  celeborn.client.push.stageEnd.timeout: 120s
  celeborn.master.ha.enabled: true
  celeborn.master.ha.ratis.raft.server.storage.dir: /mnt/celeborn_ratis
  celeborn.master.heartbeat.application.timeout: 300s
  celeborn.master.heartbeat.worker.timeout: 120s
  celeborn.master.http.port: 9098
  celeborn.metrics.enabled: true
  celeborn.metrics.prometheus.path: /metrics/prometheus
  celeborn.rpc.dispatcher.numThreads: 4
  celeborn.rpc.io.clientThreads: 64
  celeborn.rpc.io.numConnectionsPerPeer: 2
  celeborn.rpc.io.serverThreads: 64
  celeborn.shuffle.chunk.size: 8m
  celeborn.worker.fetch.io.threads: 32
  celeborn.worker.flusher.buffer.size: 256K
  celeborn.worker.http.port: 9096
  celeborn.worker.monitor.disk.enabled: false
  celeborn.worker.push.io.threads: 32
  celeborn.worker.storage.dirs: /mnt/disk1:disktype=SSD:capacity=1024Gi,/mnt/disk2:disktype=SSD:capacity=1024Gi,/mnt/disk3:disktype=SSD:capacity=1024Gi,/mnt/disk4:disktype=SSD:capacity=1024Gi

master:
  replicas: 3
  env:
  - name: CELEBORN_MASTER_MEMORY
    value: 28g
  - name: CELEBORN_MASTER_JAVA_OPTS
    value: -XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-master.out -Dio.netty.leakDetectionLevel=advanced
  - name: CELEBORN_NO_DAEMONIZE
    value: "1"
  - name: TZ
    value: Asia/Shanghai
  volumeMounts:
  - name: celeborn-ratis
    mountPath: /mnt/celeborn_ratis
  resources:
    requests:
      cpu: 7                
      memory: 28Gi        
    limits:
      cpu: 7
      memory: 28Gi
  volumes:
  - name: celeborn-ratis
    hostPath:
      path: /mnt/celeborn_ratis
      type: DirectoryOrCreate
  nodeSelector:
    celeborn.apache.org/role: master
  tolerations:
  - key: celeborn.apache.org/role
    operator: Equal
    value: master
    effect: NoSchedule

worker:
  replicas: 5
  env:
  - name: CELEBORN_WORKER_MEMORY
    value: 28g
  - name: CELEBORN_WORKER_OFFHEAP_MEMORY
    value: 28g
  - name: CELEBORN_WORKER_JAVA_OPTS
    value: -XX:-PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -Xloggc:gc-worker.out -Dio.netty.leakDetectionLevel=advanced
  - name: CELEBORN_NO_DAEMONIZE
    value: "1"
  - name: TZ
    value: Asia/Shanghai
  volumeMounts:
  - name: disk1
    mountPath: /mnt/disk1
  - name: disk2
    mountPath: /mnt/disk2
  - name: disk3
    mountPath: /mnt/disk3
  - name: disk4
    mountPath: /mnt/disk4
  resources:
    requests:
      cpu: 14
      memory: 56Gi
    limits:
      cpu: 14
      memory: 56Gi
  volumes:
  - name: disk1
    hostPath:
      path: /mnt/disk1
      type: DirectoryOrCreate
  - name: disk2
    hostPath:
      path: /mnt/disk2
      type: DirectoryOrCreate
  - name: disk3
    hostPath:
      path: /mnt/disk3
      type: DirectoryOrCreate
  - name: disk4
    hostPath:
      path: /mnt/disk4
      type: DirectoryOrCreate
  nodeSelector:
    celeborn.apache.org/role: worker
  tolerations:
  - key: celeborn.apache.org/role
    operator: Equal
    value: worker
    effect: NoSchedule

      The following table describes key parameters. For the full list, see the ConfigMaps section on the ack-celeborn page.

      Parameter settings

      Parameter

      Description

      Example

      image.registry

      The address of the image registry.

      "docker.io"

      image.repository

      The image name.

      "apache/celeborn"

      image.tag

      The image tag.

      "0.5.2"

      image.pullPolicy

      The image pull policy.

      "IfNotPresent"

      celeborn

      Celeborn configuration properties.

      		 
      {
  "celeborn.client.push.stageEnd.timeout": "120s",
  "celeborn.master.ha.enabled": true,
  "celeborn.master.ha.ratis.raft.server.storage.dir": "/mnt/celeborn_ratis",
  "celeborn.master.heartbeat.application.timeout": "300s",
  "celeborn.master.heartbeat.worker.timeout": "120s",
  "celeborn.master.http.port": 9098,
  "celeborn.metrics.enabled": true,
  "celeborn.metrics.prometheus.path": "/metrics/prometheus",
  "celeborn.rpc.dispatcher.numThreads": 4,
  "celeborn.rpc.io.clientThreads": 64,
  "celeborn.rpc.io.numConnectionsPerPeer": 2,
  "celeborn.rpc.io.serverThreads": 64,
  "celeborn.shuffle.chunk.size": "8m",
  "celeborn.worker.fetch.io.threads": 32,
  "celeborn.worker.flusher.buffer.size": "256K",
  "celeborn.worker.http.port": 9096,
  "celeborn.worker.monitor.disk.enabled": false,
  "celeborn.worker.push.io.threads": 32,
  "celeborn.worker.storage.dirs": "/mnt/disk1:disktype=SSD:capacity=1024Gi,/mnt/disk2:disktype=SSD:capacity=1024Gi,/mnt/disk3:disktype=SSD:capacity=1024Gi,/mnt/disk4:disktype=SSD:capacity=1024Gi"
}

      master.replicas

      The number of master pod replicas.

      3

      master.volumeMounts

      The volumeMounts for the master container.

      		 
      [
  {
    "mountPath": "/mnt/celeborn_ratis",
    "name": "celeborn-ratis"
  }
]

      master.volumes

      The data volumes for the master pod.

      Only the hostPath and emptyDir volume types are supported.

      		 
      [
  {
    "hostPath": {
      "path": "/mnt/celeborn_ratis",
      "type": "DirectoryOrCreate"
    },
    "name": "celeborn-ratis"
  }
]

      master.nodeSelector

      The node selector for the master pod.

      {}

      master.affinity

      The affinity rules for the master pod.

      		 
      {
  "podAntiAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "app.kubernetes.io/name",
              "operator": "In",
              "values": [
                "celeborn"
              ]
            },
            {
              "key": "app.kubernetes.io/role",
              "operator": "In",
              "values": [
                "master"
              ]
            }
          ]
        },
        "topologyKey": "kubernetes.io/hostname"
      }
    ]
  }
}

      master.tolerations

      The tolerations for the master pod.

      []

      worker.replicas

      The number of worker pod replicas.

      5

      worker.volumeMounts

      The volumeMounts for the worker container.

      		 
      [
  {
    "mountPath": "/mnt/disk1",
    "name": "disk1"
  },
  {
    "mountPath": "/mnt/disk2",
    "name": "disk2"
  },
  {
    "mountPath": "/mnt/disk3",
    "name": "disk3"
  },
  {
    "mountPath": "/mnt/disk4",
    "name": "disk4"
  }
]

      worker.volumes

      The data volumes for the worker pod.

      Only the hostPath and emptyDir volume types are supported.

      		 
      [
  {
    "capacity": "100Gi",
    "diskType": "SSD",
    "hostPath": "/mnt/disk1",
    "mountPath": "/mnt/disk1",
    "type": "hostPath"
  },
  {
    "capacity": "100Gi",
    "diskType": "SSD",
    "hostPath": "/mnt/disk2",
    "mountPath": "/mnt/disk2",
    "type": "hostPath"
  },
  {
    "capacity": "100Gi",
    "diskType": "SSD",
    "hostPath": "/mnt/disk3",
    "mountPath": "/mnt/disk3",
    "type": "hostPath"
  },
  {
    "capacity": "100Gi",
    "diskType": "SSD",
    "hostPath": "/mnt/disk4",
    "mountPath": "/mnt/disk4",
    "type": "hostPath"
  }
]

      worker.nodeSelector

      The node selector for the worker pod.

      {}

      worker.affinity

      The affinity rules for the worker pod.

      		 
      {
  "podAntiAffinity": {
    "requiredDuringSchedulingIgnoredDuringExecution": [
      {
        "labelSelector": {
          "matchExpressions": [
            {
              "key": "app.kubernetes.io/name",
              "operator": "In",
              "values": [
                "celeborn"
              ]
            },
            {
              "key": "app.kubernetes.io/role",
              "operator": "In",
              "values": [
                "worker"
              ]
            }
          ]
        },
        "topologyKey": "kubernetes.io/hostname"
      }
    ]
  }
}

      worker.tolerations

      The tolerations for the worker pod.

      []

    1. Wait for the Celeborn deployment to complete. If you encounter pod issues, see Pod troubleshooting.

      kubectl get -n celeborn statefulset

      Expected output:

      NAME              READY   AGE
celeborn-master   3/3     68s
celeborn-worker   5/5     68s

    Step 3: Build a Spark container image

    This example uses Spark 3.5.3. Create a Dockerfile with the following content to build a container image and push it to your image repository.

    ARG SPARK_IMAGE=<SPARK_IMAGE>  # Replace <SPARK_IMAGE> with your Spark base image.

FROM ${SPARK_IMAGE}

# Add dependency for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars

# Add dependency for Celeborn
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/celeborn/celeborn-client-spark-3-shaded_2.12/0.5.1/celeborn-client-spark-3-shaded_2.12-0.5.1.jar ${SPARK_HOME}/jars

    Step 4: Upload test data to OSS

    To prepare test data and upload it to OSS, see Step 1: Prepare test data and upload it to OSS.

    Step 5: Create an OSS Secret

    To create a Secret for OSS access credentials, see Step 3: Create a Secret to store OSS access credentials.

    Step 6: Submit a sample Spark job

    Create a SparkApplication manifest file named spark-pagerank.yaml with the following content. Replace <SPARK_IMAGE> with your image from Step 3: Build a Spark container image, and replace <OSS_BUCKET> and <OSS_ENDPOINT> with your OSS bucket and endpoint. For Spark configuration, see the Celeborn documentation.

    apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: <SPARK_IMAGE>                                     # The Spark image. Replace <SPARK_IMAGE> with your Spark image name.
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  - oss://<OSS_BUCKET>/data/pagerank_dataset.txt           # The input test dataset. Replace <OSS_BUCKET> with your OSS bucket name.
  - "10"                                                   # The number of iterations.
  sparkVersion: 3.5.3
  hadoopConf:
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    fs.oss.endpoint: <OSS_ENDPOINT>                        # The OSS endpoint. For example, the internal endpoint for OSS in the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com. 
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  sparkConf:
    spark.shuffle.manager: org.apache.spark.shuffle.celeborn.SparkShuffleManager
    spark.serializer: org.apache.spark.serializer.KryoSerializer
    spark.celeborn.master.endpoints: celeborn-master-0.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-1.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-2.celeborn-master-svc.celeborn.svc.cluster.local
    spark.celeborn.client.spark.shuffle.writer: hash
    spark.celeborn.client.push.replicate.enabled: "false"
    spark.sql.adaptive.localShuffleReader.enabled: "false"
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
    spark.shuffle.sort.io.plugin.class: org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
    spark.dynamicAllocation.shuffleTracking.enabled: "false"
    spark.executor.userClassPathFirst: "false"
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    envFrom:
    - secretRef:
        name: spark-oss-secret
  executor:
    instances: 2
    cores: 1
    coreLimit: "2"
    memory: 8g
    envFrom:
    - secretRef:
        name: spark-oss-secret
  restartPolicy:
    type: Never

    (Optional) Step 7: Clean up resources

    After completing this tutorial, delete these resources to avoid fees.

    Delete the Spark job:

    kubectl delete sparkapplication spark-pagerank

    Delete the Secret resource:

    kubectl delete secret spark-oss-secret

    References