All Products
Search

This Product

    Container Service for Kubernetes:Configure dynamic resource allocation for Spark jobs

    Document Center

    Container Service for Kubernetes:Configure dynamic resource allocation for Spark jobs

    Last Updated:Jun 25, 2026

    This topic describes how to configure and use the dynamic resource allocation (DRA) feature in Spark to maximize cluster resource utilization, reduce idle resources, and improve job flexibility and overall system performance.

    What is dynamic resource allocation?

    Dynamic resource allocation (DRA) is a feature in Spark that adjusts the computing resources a job uses based on its workload. If an Executor is idle for an extended period, the Driver automatically releases it and returns the resources to the cluster. Conversely, if tasks are queued for too long, the Driver requests more Executors to run them. DRA helps Spark adapt to fluctuating workloads, which prevents long runtimes due to under-provisioning and resource waste from over-provisioning.

    Allocation policy

    When you enable DRA, the Driver requests additional Executors if tasks are pending. If tasks remain pending for longer than spark.dynamicAllocation.schedulerBacklogTimeout (default: 1s), the Driver starts requesting Executors in rounds. If there are still pending tasks after each spark.dynamicAllocation.sustainedSchedulerBacklogTimeout (default: 1s), the Driver continues to request more Executors. The number of Executors requested in each round increases exponentially: 1, 2, 4, 8, and so on.

    Removal policy

    If an Executor remains idle for longer than spark.dynamicAllocation.executorIdleTimeout (default: 60s), the Driver automatically releases it to optimize resource utilization.

    Enable dynamic resource allocation

    The Spark DRA mechanism is available in Standalone, YARN, Mesos, and Kubernetes run modes but is disabled by default. To enable DRA, in addition to setting spark.dynamicAllocation.enabled to true, you must also configure the following options:

    1. Enable External Shuffle Service (ESS): Set spark.shuffle.service.enabled to true and configure an External Shuffle Service on each worker node.

    2. Enable shuffle tracking: Set spark.dynamicAllocation.shuffleTracking.enabled to true. Spark tracks the location and status of shuffle data to prevent data loss and ensure it can be recomputed if needed.

    3. Enable node decommissioning: Set spark.decommission.enabled and spark.storage.decommission.shuffleBlocks.enabled to true. This setting prompts Spark to proactively migrate shuffle blocks from a node that is being decommissioned to other available nodes.

    4. Configure a ShuffleDataIO plugin: Specify a ShuffleDataIO plugin class for spark.shuffle.sort.io.plugin.class to customize shuffle I/O operations, allowing you to write shuffle data to different storage systems.

    Configuration varies by mode:

    • Spark Standalone mode: Configure Option 1 only.

    • Spark on Kubernetes mode:

      • Without using Celeborn as a Remote Shuffle Service (RSS): Set spark.dynamicAllocation.shuffleTracking.enabled to true (Option 2).

      • When using Celeborn as an RSS:

        • For Spark 3.5.0 or later: Set spark.shuffle.sort.io.plugin.class to org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO (Option 4).

        • For Spark versions earlier than 3.5.0: Apply the corresponding patch to Spark. No additional configuration is required. For more information, see Support Spark Dynamic Allocation.

        • For Spark 3.4.0 or later: We recommend setting spark.dynamicAllocation.shuffleTracking.enabled to false to ensure idle Executors are released promptly.

    This guide demonstrates configuring DRA in Spark on Kubernetes mode for scenarios with and without Celeborn as an RSS.

    Prerequisites

    Procedure

    Without Celeborn

    In this Spark 3.5.4 example, to enable DRA without a Remote Shuffle Service (RSS), set the following parameters:

    • Enable dynamic resource allocation: Set spark.dynamicAllocation.enabled to "true".

    • Enable shuffle tracking, which allows DRA without an ESS: Set spark.dynamicAllocation.shuffleTracking.enabled to "true".

    1. Create a SparkApplication manifest file named spark-pagerank-dra.yaml with the following content:

      apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank-dra
  namespace: spark
spec:
  type: Scala
  mode: cluster
  # Replace <SPARK_IMAGE> with your Spark image.
  image: <SPARK_IMAGE>
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  # Replace <OSS_BUCKET> with your OSS bucket name.
  - oss://<OSS_BUCKET>/data/pagerank_dataset.txt
  - "10"
  sparkVersion: 3.5.4
  hadoopConf:
    # Allow access to OSS data by using the oss:// format.
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    # The OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
    # For example, the internal endpoint for OSS in the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com
    fs.oss.endpoint: <OSS_ENDPOINT>
    # Read access credentials for OSS from environment variables.
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  sparkConf:
    # ====================
    # Event Logs
    # ====================
    spark.eventLog.enabled: "true"
    spark.eventLog.dir: file:///mnt/nas/spark/event-logs
    # ====================
    # Dynamic Resource Allocation
    # ====================
    # Enable dynamic resource allocation.
    spark.dynamicAllocation.enabled: "true"
    # Enable shuffle tracking to allow dynamic resource allocation without relying on an ESS.
    spark.dynamicAllocation.shuffleTracking.enabled: "true"
    # The initial number of Executors.
    spark.dynamicAllocation.initialExecutors: "1"
    # The minimum number of Executors.
    spark.dynamicAllocation.minExecutors: "0"
    # The maximum number of Executors.
    spark.dynamicAllocation.maxExecutors: "5"
    # The idle timeout for an Executor. If the timeout is exceeded, the Executor is released.
    spark.dynamicAllocation.executorIdleTimeout: 60s
    # The idle timeout for an Executor that has cached data blocks. The default is infinity, which means it is not released.
    # spark.dynamicAllocation.cachedExecutorIdleTimeout:
    # When pending tasks exceed this timeout, more Executors are requested.
    spark.dynamicAllocation.schedulerBacklogTimeout: 1s
    # The interval at which subsequent rounds of Executor requests are made.
    spark.dynamicAllocation.sustainedSchedulerBacklogTimeout: 1s
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-oss-secret
    volumeMounts:
    - name: nas
      mountPath: /mnt/nas
    serviceAccount: spark-operator-spark
  executor:
    cores: 1
    coreLimit: "2"
    memory: 8g
    envFrom:
    - secretRef:
        name: spark-oss-secret
    volumeMounts:
    - name: nas
      mountPath: /mnt/nas
  volumes:
  - name: nas
    persistentVolumeClaim:
      claimName: nas-pvc
  restartPolicy:
    type: Never
      Note

      The Spark image used in this job must include the Hadoop OSS SDK dependency. You can use the following Dockerfile to build the image and push it to your own image repository:

      ARG SPARK_IMAGE=spark:3.5.4
FROM ${SPARK_IMAGE}
# Add dependency for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars

    2. Run the following command to submit the Spark job.

      kubectl apply -f spark-pagerank-dra.yaml

      Expected output:

      sparkapplication.sparkoperator.k8s.io/spark-pagerank-dra created

    3. View the Driver logs.

      kubectl logs -n spark spark-pagerank-dra-driver | grep -a2 -b2 "Going to request"

      Expected output:

      3544-25/01/16 03:26:04 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
3674-25/01/16 03:26:06 INFO Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
3848:25/01/16 03:26:06 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 1, known: 0, sharedSlotFromPendingPods: 2147483647.
4026-25/01/16 03:26:06 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs
4106-25/01/16 03:26:06 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
--
10410-25/01/16 03:26:15 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (192.168.95.190, executor 1, partition 0, PROCESS_LOCAL, 9807 bytes)
10558-25/01/16 03:26:15 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.95.190:34327 (size: 12.5 KiB, free: 4.6 GiB)
10690:25/01/16 03:26:16 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
10868-25/01/16 03:26:16 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 2 for resource profile id: 0)
11030-25/01/16 03:26:16 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs

      The log shows that the Driver requested one Executor at 03:26:06 and another at 03:26:16.

    4. Access the Spark History Server to view the job. For instructions, see Access the Spark History Server Web UI.

      The job's event timeline also shows that two Executors were created sequentially.

    With Celeborn

    In this Spark 3.5.4 example, to enable DRA when using Celeborn as an RSS, set the following parameters:

    • Enable dynamic resource allocation: Set spark.dynamicAllocation.enabled to "true".

    • Configure support for dynamic resource allocation (for Spark 3.5.0 or later): Set spark.shuffle.sort.io.plugin.class to org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO.

    • Ensure idle Executors are released promptly: Set spark.dynamicAllocation.shuffleTracking.enabled to "false".

    1. Create a SparkApplication manifest file named spark-pagerank-celeborn-dra.yaml with the following content:

      apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pagerank-celeborn-dra
  namespace: spark
spec:
  type: Scala
  mode: cluster
  # Replace <SPARK_IMAGE> with your Spark image. This image must include dependencies for Hadoop OSS and Celeborn.
  image: <SPARK_IMAGE>
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
  mainClass: org.apache.spark.examples.SparkPageRank
  arguments:
  # Replace <OSS_BUCKET> with your OSS bucket name.
  - oss://<OSS_BUCKET>/data/pagerank_dataset.txt
  - "10"
  sparkVersion: 3.5.4
  hadoopConf:
    # Allow access to OSS data by using the oss:// format.
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    # The OSS endpoint. Replace <OSS_ENDPOINT> with your OSS endpoint.
    # For example, the internal endpoint for OSS in the China (Beijing) region is oss-cn-beijing-internal.aliyuncs.com
    fs.oss.endpoint: <OSS_ENDPOINT>
    # Read access credentials for OSS from environment variables.
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  sparkConf:
    # ====================
    # Event Logs
    # ====================
    spark.eventLog.enabled: "true"
    spark.eventLog.dir: file:///mnt/nas/spark/event-logs
    # ====================
    # Celeborn
    # Ref: https://github.com/apache/celeborn/blob/main/README.md#spark-configuration
    # ====================
    # Shuffle manager class name changed in 0.3.0:
    #    before 0.3.0: `org.apache.spark.shuffle.celeborn.RssShuffleManager`
    #    since 0.3.0: `org.apache.spark.shuffle.celeborn.SparkShuffleManager`
    spark.shuffle.manager: org.apache.spark.shuffle.celeborn.SparkShuffleManager
    # Must use kryo serializer because java serializer do not support relocation.
    spark.serializer: org.apache.spark.serializer.KryoSerializer
    # This must be configured based on the number of Celeborn master replicas.
    spark.celeborn.master.endpoints: celeborn-master-0.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-1.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-2.celeborn-master-svc.celeborn.svc.cluster.local
    # options: hash, sort
    # Hash shuffle writer uses (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
    # Sort shuffle writer uses less memory than hash shuffle writer. If your shuffle partition count is large, try using the sort hash writer.
    spark.celeborn.client.spark.shuffle.writer: hash
    # We recommend setting `spark.celeborn.client.push.replicate.enabled` to true to enable server-side data replication.
    # If you have only one worker, this setting must be false.
    # If your Celeborn is using HDFS, it's recommended to set this to false.
    spark.celeborn.client.push.replicate.enabled: "false"
    # Support for Spark AQE is only tested under Spark 3.
    spark.sql.adaptive.localShuffleReader.enabled: "false"
    # We recommend enabling AQE support to gain better performance.
    spark.sql.adaptive.enabled: "true"
    spark.sql.adaptive.skewJoin.enabled: "true"
    # For Spark 3.5.0 or later, configure this option to support dynamic resource allocation.
    spark.shuffle.sort.io.plugin.class: org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
    spark.executor.userClassPathFirst: "false"
    # ====================
    # Dynamic Resource Allocation
    # Ref: https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
    # ====================
    # Enable dynamic resource allocation.
    spark.dynamicAllocation.enabled: "true"
    # Enable shuffle tracking to allow dynamic resource allocation without relying on an ESS.
    # When using Celeborn as an RSS with Spark 3.4.0 or later, we strongly recommend you disable this option.
    spark.dynamicAllocation.shuffleTracking.enabled: "false"
    # The initial number of Executors.
    spark.dynamicAllocation.initialExecutors: "1"
    # The minimum number of Executors.
    spark.dynamicAllocation.minExecutors: "0"
    # The maximum number of Executors.
    spark.dynamicAllocation.maxExecutors: "5"
    # The idle timeout for an Executor. If the timeout is exceeded, the Executor is released.
    spark.dynamicAllocation.executorIdleTimeout: 60s
    # The idle timeout for an Executor that has cached data blocks. The default is infinity, which means it is not released.
    # spark.dynamicAllocation.cachedExecutorIdleTimeout:
    # When pending tasks exceed this timeout, more Executors are requested.
    spark.dynamicAllocation.schedulerBacklogTimeout: 1s
    # The interval at which subsequent rounds of Executor requests are made.
    spark.dynamicAllocation.sustainedSchedulerBacklogTimeout: 1s
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-oss-secret
    volumeMounts:
    - name: nas
      mountPath: /mnt/nas
    serviceAccount: spark-operator-spark
  executor:
    cores: 1
    coreLimit: "1"
    memory: 4g
    envFrom:
    - secretRef:
        name: spark-oss-secret
    volumeMounts:
    - name: nas
      mountPath: /mnt/nas
  volumes:
  - name: nas
    persistentVolumeClaim:
      claimName: nas-pvc
  restartPolicy:
    type: Never
      Note

      The Spark image used in the preceding example job must include dependencies for the Hadoop OSS SDK and Celeborn. You can use the following Dockerfile to build the image and push it to your own image repository:

      ARG SPARK_IMAGE=spark:3.5.4
FROM ${SPARK_IMAGE}
# Add dependency for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars
# Add dependency for Celeborn
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/celeborn/celeborn-client-spark-3-shaded_2.12/0.5.3/celeborn-client-spark-3-shaded_2.12-0.5.3.jar ${SPARK_HOME}/jars

    2. Run the following command to submit the Spark job.

      kubectl apply -f spark-pagerank-celeborn-dra.yaml

      Expected output:

      sparkapplication.sparkoperator.k8s.io/spark-pagerank-celeborn-dra created

    3. View the Driver logs.

      kubectl logs -n spark spark-pagerank-celeborn-dra-driver | grep -a2 -b2 "Going to request"

      Expected output:

      3544-25/01/16 03:51:28 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
3674-25/01/16 03:51:30 INFO Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
3848:25/01/16 03:51:30 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 1, known: 0, sharedSlotFromPendingPods: 2147483647.
4026-25/01/16 03:51:30 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs
4106-25/01/16 03:51:30 INFO CelebornShuffleDataIO: Loading CelebornShuffleDataIO
--
11796-25/01/16 03:51:41 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (192.168.95.163, executor 1, partition 0, PROCESS_LOCAL, 9807 bytes)
11944-25/01/16 03:51:42 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.95.163:37665 (size: 13.3 KiB, free: 2.1 GiB)
12076:25/01/16 03:51:42 INFO ExecutorPodsAllocator: Going to request 1 executors from Kubernetes for ResourceProfile Id: 0, target: 2, known: 1, sharedSlotFromPendingPods: 2147483647.
12254-25/01/16 03:51:42 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 2 for resource profile id: 0)
12416-25/01/16 03:51:42 INFO ExecutorPodsAllocator: Found 0 reusable PVCs from 0 PVCs

      The log shows that the Driver requested an Executor at 03:51:30 and another at 03:51:42.

    4. Access the Spark History Server to view the job. For instructions, see Access the Spark History Server Web UI.

      The job's event timeline also shows that two Executors were created sequentially.