Use elastic container instances to run Spark jobs - Container Service for Kubernetes

This topic describes how to run Spark jobs on ACK using Elastic Container Instance (ECI) resources. By configuring appropriate scheduling strategies, you can create ECI Pods on demand and pay only for resources consumed, reducing idle costs and running Spark jobs more economically.

Prerequisites

You need:

ack-spark-operator installed.
ack-virtual-node deployed in the cluster (exposes ECI capacity as virtual nodes).
kubectl connected to the ACK cluster.

How it works

When ack-virtual-node is deployed, each ECI node has a virtual-kubelet.io/provider=alibabacloud:NoSchedule taint. Without a matching toleration, the scheduler skips ECI nodes and places all pods on ECS. To route Spark pods to ECI, add tolerations and node affinity rules to the SparkApplication spec.

ECI runs each container in a lightweight virtual sandbox, fully isolating pods. With pay-as-you-go billing, you pay only for the CPU and memory each pod consumes while running—not for idle node capacity.

Driver vs. executor placement

Driver and executor pods fail differently, which affects placement:

If an executor fails, Spark automatically replaces it. Executors are stateless and fault-tolerant.
If the driver fails, the entire job fails and restarts from the beginning. Keep the driver stable.

Keep the driver on reliable ECS nodes and route executors to lower-cost ECI or preemptible instances when cost matters.

ECI characteristics for Spark workloads:

Characteristic	Value
Scale	50,000+ pods in an ACK Serverless cluster; no extra configuration
Provisioning speed	Thousands of pods in seconds
Billing	Pay-as-you-go; preemptible instances available for further cost reduction

Choose a scheduling strategy

Three strategies cover common deployment scenarios:

Strategy	When to use	Driver placement	Executor placement
ECS only	Predictable, steady workloads	ECS	ECS
ECI only	Batch or burst jobs that need full elasticity	ECI	ECI
ECS-first with ECI fallback	Normal load on ECS; auto-expand to ECI during peaks	ECS (preferred)	ECS (preferred), ECI (overflow)

For fine-grained control, such as capping pods per resource type, use a ResourcePolicy.

Schedule Spark jobs on ECS and ECI nodes

All examples use SparkApplication (CRD sparkoperator.k8s.io/v1beta2) with these base settings:

Field	Value
Image	`registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2`
`mainClass`	`org.apache.spark.examples.SparkPi`
`sparkVersion`	`3.5.2`
`driver.serviceAccount`	`spark-operator-spark`

ECS only

No toleration or affinity required. The default ECI taint prevents the scheduler from placing pods there.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-ecs-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 2
    cores: 2
    memory: 4g

Expected behavior: All driver and executor pods schedule to ECS nodes. If ECS capacity is insufficient, pods stay Pending until capacity is available.

ECI only

Add a toleration for the default ECI taint and a requiredDuringSchedulingIgnoredDuringExecution affinity rule to pin pods to ECI nodes. Apply both to the driver and executor specs.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # Pin driver to ECI nodes only.
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # Tolerate the default ECI taint.
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # Pin executors to ECI nodes only.
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    # Tolerate the default ECI taint.
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

Expected behavior: All pods schedule to ECI nodes and the job runs in fully serverless mode. If regional ECI capacity is limited, pods stay Pending until resources are available.

ECS-first with ECI fallback

Use preferredDuringSchedulingIgnoredDuringExecution affinity to favor ECS while allowing ECI overflow. Add the ECI toleration so the scheduler can place pods on ECI when ECS is full.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-ecs-first
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        # Prefer ECS nodes; fall back to ECI if ECS capacity is exhausted.
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: type
              operator: NotIn
              values:
              - virtual-kubelet
    tolerations:
    # Allow scheduling to ECI nodes when needed.
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        # Prefer ECS nodes; fall back to ECI if ECS capacity is exhausted.
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
            - key: type
              operator: NotIn
              values:
              - virtual-kubelet
    tolerations:
    # Allow scheduling to ECI nodes when needed.
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

Expected behavior: Pods land on ECS when capacity is available. When ECS is full, overflow pods schedule to ECI. This avoids job queuing during peaks without permanently reserving ECI resources.

See Configure resource allocation based on ECS instances and elastic container instances for taints, tolerations, and node affinity.

Configure priority-based resource scheduling

Use a ResourcePolicy to define scheduling units with per-unit pod caps. The ACK scheduler fills units in priority order during scale-out and removes pods in reverse order during scale-in. See Configure priority-based resource scheduling.

Create resourcepolicy.yaml for Spark Operator pods in the default namespace (max 2 on AMD64 ECS, then 3 on ECI):

apiVersion: scheduling.alibabacloud.com/v1alpha1
kind: ResourcePolicy
metadata:
  name: sparkapplication-resource-policy
  namespace: default                      # Applies only to pods in this namespace.
spec:
  ignorePreviousPod: true
  ignoreTerminatingPod: false
  matchLabelKeys:
  - sparkoperator.k8s.io/submission-id    # Group pods by Spark job submission ID.
  preemptPolicy: AfterAllUnits            # Attempt preemption only after all units are exhausted.
  selector:
    sparkoperator.k8s.io/launched-by-spark-operator: "true"
  strategy: prefer
  units:
  - max: 2                               # Up to 2 pods on AMD64 ECS nodes (first priority).
    resource: ecs
    nodeSelector:
      kubernetes.io/arch: amd64
  - max: 3                               # Up to 3 pods on ECI (second priority).
    resource: eci

Apply the ResourcePolicy:
```
kubectl apply -f resourcepolicy.yaml
```

Create spark-pi.yaml. This SparkApplication requests 1 driver and 5 executors, all tolerating the ECI taint so the ResourcePolicy can distribute them across both unit types.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    tolerations:
    - key: virtual-kubelet.io/provider    # Allow the driver to be scheduled to ECI if needed.
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    instances: 5
    cores: 1
    coreLimit: 1200m
    memory: 512m
    tolerations:
    - key: virtual-kubelet.io/provider    # Allow executors to be scheduled to ECI.
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

Submit the Spark job:
```
kubectl apply -f spark-pi.yaml
```

Verify scheduling results:

kubectl get pods -o wide -l sparkoperator.k8s.io/app-name=spark-pi

Expected output:

NAME                                        READY   STATUS      RESTARTS   AGE       IP                  NODE
spark-pi-34c0998f9f832e61-exec-1            1/1     Running     0          28s       192.XXX.XX.34       cn-beijing.192.XXX.XX.250
spark-pi-34c0998f9f832e61-exec-2            1/1     Running     0          28s       192.XXX.XX.87       virtual-kubelet-cn-beijing-i
spark-pi-34c0998f9f832e61-exec-3            1/1     Running     0          28s       192.XXX.XX.88       virtual-kubelet-cn-beijing-i
spark-pi-34c0998f9f832e61-exec-4            1/1     Running     0          28s       192.XXX.XX.86       virtual-kubelet-cn-beijing-i
spark-pi-34c0998f9f832e61-exec-5            0/1     Pending     0          28s       <none>              <none>
spark-pi-driver                             1/1     Running     0          34s       192.XXX.XX.37       cn-beijing.192.XXX.XXX.250

Result: The driver and exec-1 land on the AMD64 ECS unit (max 2 pods). exec-2, exec-3, and exec-4 land on the ECI unit (max 3 pods). exec-5 stays Pending because both units reached their pod cap.

Accelerate image pulling with ImageCache

Pulling a large Spark image on each pod startup adds latency. ECI ImageCache pre-caches images on the underlying infrastructure, cut pod startup from ~100 seconds to near-instant on a cache hit. See Use ImageCache to accelerate the creation of elastic container instances.

Compare startup with and without an image cache

Without a cache, kubectl describe pod spark-pi-driver after submitting a SparkApplication shows:

Events:
  ...
  Warning  ImageCacheMissed       24m   EciService         [eci.imagecache]Missed image cache.
  Normal   ImageCacheAutoCreated  24m   EciService         [eci.imagecache]Image cache imc-2zeXXXXXXXXXXXXXXXXX is auto created
  Normal   Pulling                24m   kubelet            Pulling image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2"
  Normal   Pulled                 23m   kubelet            Successfully pulled image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2" in 1m41.289s (1m41.289s including waiting)
  ...

The image took about 100 seconds to pull. ECI automatically created a cache for later runs.

With a cache hit, the events show:

Events:
  ...
  Normal  SuccessfulHitImageCache  23s   EciService         [eci.imagecache]Successfully hit image cache imc-2zeXXXXXXXXXXXXXXXXX, eci will be scheduled with this image cache.
  Normal  Pulled                   4s    kubelet            Container image "registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2" already present on machine
  ...

No image pull was required.

Specify an image cache ID

Add the k8s.aliyun.com/eci-image-snapshot-id annotation to the driver and executor specs to pin a specific image cache:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    annotations:
      k8s.aliyun.com/eci-image-snapshot-id: imc-2zeXXXXXXXXXXXXXXXXX  # Image cache ID.
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    annotations:
      k8s.aliyun.com/eci-image-snapshot-id: imc-2zeXXXXXXXXXXXXXXXXX  # Image cache ID.
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

Enable automatic image cache creation and matching

To let ECI manage cache creation and matching automatically—without a cache ID—set the k8s.aliyun.com/eci-image-cache annotation to "true" on the driver and executor:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-eci-only
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  arguments:
  - "5000"
  sparkVersion: 3.5.2
  driver:
    annotations:
      k8s.aliyun.com/eci-image-cache: "true"  # Enable automatic image cache creation and matching.
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule
  executor:
    annotations:
      k8s.aliyun.com/eci-image-cache: "true"  # Enable automatic image cache creation and matching.
    instances: 2
    cores: 2
    memory: 4g
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: type
              operator: In
              values:
              - virtual-kubelet
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Equal
      value: alibabacloud
      effect: NoSchedule

On first run, ECI creates a cache automatically. Later runs match the cache and skip the image pull.

Apply in production

Use ECS-first with ECI fallback for production jobs. Keeps latency-sensitive work on persistent ECS nodes while ECI absorbs burst traffic without pre-provisioning extra nodes.
Run the driver on ECS, executors on ECI. A driver failure restarts the entire job. Keep the driver on stable ECS nodes and route stateless executors to lower-cost ECI or preemptible instances.
Enable automatic image caching for repeated jobs. The first run creates the cache; later runs skip the image pull, cut startup from ~100 seconds to near-instant.
Use ResourcePolicy when pod distribution across node types must be precise. The max field per unit caps pods, useful for limiting ECI spend per job submission.

Container Service for Kubernetes:Use elastic container instances to run Spark jobs