Run Spark on ACK with integrated Alibaba Cloud storage, observability, and elastic scaling via the Spark Operator.
Billing
Installing Spark-related ACK components (ack-spark-operator, ack-spark-history-server, and others) is free. Standard ACK cluster fees — cluster management fees and associated cloud resource fees — apply. See Billing overview.
Additional fees from other cloud products may apply. For example, SLS charges for log collection, and OSS or NAS charges apply for data read and write operations by Spark jobs.
Getting started
Running Spark jobs on ACK follows a layered setup: start with the basics, add observability, then tune for performance.
Prerequisites
Ensure you have:
-
A running ACK cluster with kubectl access configured
-
Sufficient permissions to create and manage pods in your cluster. Verify:
kubectl auth can-i create pods
-
A dedicated namespace for Spark jobs (this guide uses spark):
kubectl create namespace spark
-
A service account (such as spark-operator-spark) with RBAC permissions to create, list, and delete executor pods and services in the job namespace.
Basic usage
Step 1: Build a Spark container image
Use the open-source Spark image, or customize it with dependencies such as OSS support or Celeborn RSS. This Dockerfile adds the dependencies used in this guide.
Expand to view the sample Dockerfile
ARG SPARK_IMAGE=spark:3.5.4
FROM ${SPARK_IMAGE}
# Add dependency for Hadoop Aliyun OSS support
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars
# Add dependency for log4j-layout-template-json
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-layout-template-json/2.24.1/log4j-layout-template-json-2.24.1.jar ${SPARK_HOME}/jars
# Add dependency for Celeborn
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/celeborn/celeborn-client-spark-3-shaded_2.12/0.5.3/celeborn-client-spark-3-shaded_2.12-0.5.3.jar ${SPARK_HOME}/jars
Build the image and push it to your image repository, then reference it in your SparkApplication resources.
Step 2: Deploy Spark Operator and run your first job
Deploy the ack-spark-operator component and set spark.jobNamespaces=["spark"] so it only watches jobs in the spark namespace.
This minimal SparkApplication runs SparkPi to verify Spark Operator is working:
Expand to view the sample Spark job
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pi
namespace: spark # Must be in the namespace list specified by spark.jobNamespaces
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with your own Spark container image
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
- "5000"
sparkVersion: 3.5.4
driver:
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-driver
serviceAccount: spark-operator-spark
executor:
instances: 1
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-executor
restartPolicy:
type: Never
restartPolicy: type: Never suits batch jobs that should not retry on failure. Set to OnFailure (with onFailureRetries and onFailureRetryInterval) for production pipelines that require automatic retries.
See Use Spark Operator to run Spark jobs.
Step 3: Read and write OSS data
Spark jobs access OSS via Hadoop Aliyun SDK, Hadoop AWS SDK, or JindoSDK. Include the required dependencies in your container image and configure the Hadoop parameters.
Expand to view the sample code
This example runs SparkPageRank with input data from OSS. Upload your test dataset first. See Read and write OSS data in Spark jobs.
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: spark
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with your own Spark image
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPageRank
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
# Replace <OSS_BUCKET> with your OSS bucket name
- oss://<OSS_BUCKET>/data/pagerank_dataset.txt
# Number of iterations
- "10"
sparkVersion: 3.5.4
hadoopConf:
fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
# Replace <OSS_ENDPOINT> with the OSS endpoint, for example oss-cn-beijing-internal.aliyuncs.com
fs.oss.endpoint: <OSS_ENDPOINT>
fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
driver:
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-driver
envFrom:
# Read OSS credentials from a Kubernetes Secret
- secretRef:
name: spark-oss-secret
serviceAccount: spark-operator-spark
executor:
instances: 2
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-executor
envFrom:
- secretRef:
name: spark-oss-secret
restartPolicy:
type: Never
See Read and write OSS data in Spark jobs.
Observability
Deploy Spark History Server
Deploy ack-spark-history-server in the spark namespace. It reads Spark event logs from a storage backend (PVC, OSS/OSS-HDFS, or HDFS) and exposes them through a web UI.
This example configures Spark History Server to read event logs from NAS at /spark/event-logs:
Expand to view the sample configuration
# Spark configuration
sparkConf:
spark.history.fs.logDirectory: file:///mnt/nas/spark/event-logs
# Environment variables
env:
- name: SPARK_DAEMON_MEMORY
value: 7g
# Data volume
volumes:
- name: nas
persistentVolumeClaim:
claimName: nas-pvc
# Data volume mount
volumeMounts:
- name: nas
subPath: spark/event-logs
mountPath: /mnt/nas/spark/event-logs
# Adjust resource size based on the number and scale of Spark jobs
resources:
requests:
cpu: 2
memory: 8Gi
limits:
cpu: 2
memory: 8Gi
Mount the same NAS file system in your Spark jobs and set spark.eventLog.dir to the same path. Example job with event logging:
Expand to view the sample Spark job
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pi
namespace: spark
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with your Spark image
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
- "5000"
sparkVersion: 3.5.4
sparkConf:
spark.eventLog.enabled: "true"
spark.eventLog.dir: file:///mnt/nas/spark/event-logs
driver:
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-driver
volumeMounts:
- name: nas
subPath: spark/event-logs
mountPath: /mnt/nas/spark/event-logs
volumes:
- name: nas
persistentVolumeClaim:
claimName: nas-pvc
serviceAccount: spark-operator-spark
executor:
instances: 1
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-executor
restartPolicy:
type: Never
See View Spark jobs with Spark History Server.
Collect Spark logs with Simple Log Service
For clusters with many jobs, use SLS to centrally collect stdout and stderr logs from all Spark containers.
Expand to view the sample job
This example configures SLS to collect logs from /opt/spark/logs/*.log in Spark containers.
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pi
namespace: spark
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with the Spark image built in step one
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
- "5000"
sparkVersion: 3.5.4
# Read log4j2.properties from the specified ConfigMap
sparkConfigMap: spark-log-conf
sparkConf:
spark.eventLog.enabled: "true"
spark.eventLog.dir: file:///mnt/nas/spark/event-logs
driver:
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-driver
volumeMounts:
- name: nas
subPath: spark/event-logs
mountPath: /mnt/nas/spark/event-logs
serviceAccount: spark-operator-spark
volumes:
- name: nas
persistentVolumeClaim:
claimName: nas-pvc
executor:
instances: 1
cores: 1
coreLimit: 1200m
memory: 512m
template:
spec:
containers:
- name: spark-kubernetes-executor
restartPolicy:
type: Never
See Collect Spark job logs with SLS.
Performance optimization
Improve shuffle performance with RSS
Shuffle operations involve heavy disk I/O, serialization, and network I/O — common causes of OOM errors and fetch failures. Configure Apache Celeborn as the Remote Shuffle Service for storage-compute separation and improved shuffle stability.
Deploy ack-celeborn first, then reference it in your job configuration. All examples use spark.shuffle.manager: org.apache.spark.shuffle.celeborn.SparkShuffleManager and spark.celeborn.master.endpoints pointing to the Celeborn master pods.
Expand to view the sample code
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: spark
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with your Spark image
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPageRank
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
- oss://<OSS_BUCKET>/data/pagerank_dataset.txt
- "10"
sparkVersion: 3.5.4
hadoopConf:
fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
fs.oss.endpoint: <OSS_ENDPOINT>
fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
sparkConfigMap: spark-log-conf
sparkConf:
spark.eventLog.enabled: "true"
spark.eventLog.dir: file:///mnt/nas/spark/event-logs
# Celeborn RSS configuration
spark.shuffle.manager: org.apache.spark.shuffle.celeborn.SparkShuffleManager
# KryoSerializer is required because Java serializer does not support relocation
spark.serializer: org.apache.spark.serializer.KryoSerializer
# Configure based on the number of Celeborn master replicas
spark.celeborn.master.endpoints: celeborn-master-0.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-1.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-2.celeborn-master-svc.celeborn.svc.cluster.local
spark.celeborn.client.spark.shuffle.writer: hash
spark.celeborn.client.push.replicate.enabled: "false"
spark.sql.adaptive.localShuffleReader.enabled: "false"
spark.sql.adaptive.enabled: "true"
spark.sql.adaptive.skewJoin.enabled: "true"
spark.shuffle.sort.io.plugin.class: org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
spark.dynamicAllocation.shuffleTracking.enabled: "false"
spark.executor.userClassPathFirst: "false"
driver:
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-driver
envFrom:
- secretRef:
name: spark-oss-secret
volumeMounts:
- name: nas
subPath: spark/event-logs
mountPath: /mnt/nas/spark/event-logs
volumes:
- name: nas
persistentVolumeClaim:
claimName: nas-pvc
serviceAccount: spark-operator-spark
executor:
instances: 2
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-executor
envFrom:
- secretRef:
name: spark-oss-secret
restartPolicy:
type: Never
See Use Celeborn as RSS in Spark jobs.
Define elastic resource scheduling priority
Use ECI-based pods with a ResourcePolicy to run Spark jobs on demand and pay only for actual usage. The ACK scheduler assigns pods to ECS or ECI based on the configured strategy — no SparkApplication changes required.
Expand to view the sample elastic policy
This example prioritizes ECS resources (up to 10 pods) and falls back to elastic container instances (up to 10 pods) when ECS capacity is insufficient:
apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
name: spark
namespace: spark
spec:
# Apply this strategy to pods launched by Spark Operator
selector:
sparkoperator.k8s.io/launched-by-spark-operator: "true"
strategy: prefer
units:
# First: use ECS resources, up to 10 pods
- resource: ecs
max: 10
podLabels:
k8s.aliyun.com/resource-policy-wait-for-ecs-scaling: "true"
nodeSelector:
node.alibabacloud.com/instance-charge-type: PostPaid
# Second: use ECI resources, up to 10 pods
- resource: eci
max: 10
ignorePreviousPod: false
ignoreTerminatingPod: true
preemptPolicy: AfterAllUnits
whenTryNextUnits:
policy: TimeoutOrExceedMax
# Wait up to 30 seconds for ECS autoscaling before falling back to ECI
timeout: 30s
See Use elastic container instances to run Spark jobs.
Configure Dynamic Resource Allocation
Dynamic Resource Allocation (DRA) adjusts executor count based on workload, preventing resource starvation and waste. Example DRA configuration with Celeborn RSS:
Expand to view the sample job
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
name: spark-pagerank
namespace: spark
spec:
type: Scala
mode: cluster
# Replace <SPARK_IMAGE> with your Spark image
image: <SPARK_IMAGE>
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPageRank
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
arguments:
- oss://<OSS_BUCKET>/data/pagerank_dataset.txt
- "10"
sparkVersion: 3.5.4
hadoopConf:
fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
fs.oss.endpoint: <OSS_ENDPOINT>
fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
sparkConfigMap: spark-log-conf
sparkConf:
# ====================
# Event log
# ====================
spark.eventLog.enabled: "true"
spark.eventLog.dir: file:///mnt/nas/spark/event-logs
# ====================
# Celeborn
# Ref: https://github.com/apache/celeborn/blob/main/README.md#spark-configuration
# ====================
# Shuffle manager class name changed in 0.3.0:
# before 0.3.0: `org.apache.spark.shuffle.celeborn.RssShuffleManager`
# since 0.3.0: `org.apache.spark.shuffle.celeborn.SparkShuffleManager`
spark.shuffle.manager: org.apache.spark.shuffle.celeborn.SparkShuffleManager
# Must use KryoSerializer because Java serializer does not support relocation
spark.serializer: org.apache.spark.serializer.KryoSerializer
# Configure based on the number of Celeborn master replicas
spark.celeborn.master.endpoints: celeborn-master-0.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-1.celeborn-master-svc.celeborn.svc.cluster.local,celeborn-master-2.celeborn-master-svc.celeborn.svc.cluster.local
# options: hash, sort
# Hash shuffle writer uses (partition count) * (celeborn.push.buffer.max.size) * (spark.executor.cores) memory.
# Sort shuffle writer uses less memory — use it when partition count is large.
spark.celeborn.client.spark.shuffle.writer: hash
# Enable server-side data replication if you have more than one worker
# If your Celeborn is using HDFS, set this to false
spark.celeborn.client.push.replicate.enabled: "false"
spark.sql.adaptive.localShuffleReader.enabled: "false"
spark.sql.adaptive.enabled: "true"
spark.sql.adaptive.skewJoin.enabled: "true"
# Required for Spark >= 3.5.0 to support dynamic resource allocation with Celeborn
spark.shuffle.sort.io.plugin.class: org.apache.spark.shuffle.celeborn.CelebornShuffleDataIO
spark.executor.userClassPathFirst: "false"
# ====================
# Dynamic resource allocation
# Ref: https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
# ====================
spark.dynamicAllocation.enabled: "true"
# Disable shuffle tracking when using Celeborn as RSS (Spark >= 3.4.0)
spark.dynamicAllocation.shuffleTracking.enabled: "false"
spark.dynamicAllocation.initialExecutors: "3"
spark.dynamicAllocation.minExecutors: "0"
spark.dynamicAllocation.maxExecutors: "10"
# Release idle executors after 60 seconds
spark.dynamicAllocation.executorIdleTimeout: 60s
# Release executors that have cached data blocks after the specified timeout (default: infinity)
# spark.dynamicAllocation.cachedExecutorIdleTimeout:
# Request additional executors when scheduling backlog exceeds 1 second
spark.dynamicAllocation.schedulerBacklogTimeout: 1s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout: 1s
driver:
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-driver
envFrom:
- secretRef:
name: spark-oss-secret
volumeMounts:
- name: nas
subPath: spark/event-logs
mountPath: /mnt/nas/spark/event-logs
volumes:
- name: nas
persistentVolumeClaim:
claimName: nas-pvc
serviceAccount: spark-operator-spark
executor:
cores: 1
coreLimit: "1"
memory: 4g
template:
spec:
containers:
- name: spark-kubernetes-executor
envFrom:
- secretRef:
name: spark-oss-secret
restartPolicy:
type: Never
See Configure dynamic resource allocation for Spark jobs.
Use Fluid to accelerate data access
If your data is in a remote data center or you encounter data access bottlenecks, use Fluid's distributed cache to accelerate reads.
See Use Fluid to accelerate data access for Spark applications.
