Run Spark jobs with Spark Operator - Container Service for Kubernetes

Install Spark Operator on ACK to manage the full lifecycle of Spark jobs with declarative Kubernetes manifests.

Prerequisites

An ACK Pro cluster or ACK Serverless Pro cluster running Kubernetes 1.24 or later. See Create an ACK managed cluster, Create an ACK Serverless cluster, and Manually upgrade ACK clusters.

A kubectl client is connected to the ACK cluster. For more information, see Connect to an ACK cluster using kubectl.

How it works

Spark Operator automates Spark job lifecycle on Kubernetes using CustomResourceDefinition (CRD) resources such as SparkApplication and ScheduledSparkApplication. It leverages native Kubernetes features like auto-scaling, health checks, and resource management. ACK provides ack-spark-operator, based on kubeflow/spark-operator. See Spark Operator | Kubeflow.

Benefits:

Simplified management: Automate Spark job deployment and lifecycle with declarative Kubernetes configurations.
Multi-tenancy support: Use Kubernetes namespaces and resource quotas for resource isolation. Use node selection to run Spark workloads on dedicated resources.
Elastic resource provisioning: Scale with elastic resources such as Elastic Container Instance (ECI) or elastic node pools during peak loads to balance performance and cost.

Use cases:

Data analytics: Use Spark for interactive analytics and data cleansing.
Batch computing: Run scheduled batch jobs to process large-scale datasets.
Real-time processing: Spark Streaming enables real-time data stream processing.

Procedure overview

The workflow covers deploying Spark Operator, submitting jobs, monitoring execution, and managing job lifecycle.

Deploy the ack-spark-operator component: Install Spark Operator in your ACK cluster.
Submit a Spark job: Create and submit a Spark job manifest.
Monitor the Spark job: Check job status, pod status, and logs.
Access the Spark web UI: View job execution details in the browser.
Update the Spark job: Modify and reapply the job manifest.
Delete the Spark job: Remove the job and release resources.

Step 1: Deploy the ack-spark-operator component

Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.
On the Marketplace page, click the App Catalog tab, then search for and select ack-spark-operator.
On the ack-spark-operator page, click Deploy.
In the Create panel, select a cluster and namespace, and then click Next.

On the Parameters page, configure the parameters and then click OK.

Key parameters are listed below. For a complete list, see the ConfigMaps tab on the ack-spark-operator page.

Parameter	Description	Default
`controller.replicas`	The number of controller replicas.	1
`webhook.replicas`	The number of webhook replicas.	1
`spark.jobNamespaces`	Namespaces where Spark jobs can run. Empty string allows all namespaces. Separate multiple values with commas (`,`).	`["default"]` (default) `[""]` (all namespaces) `["ns1","ns2","ns3"]` (multiple namespaces)
`spark.serviceAccount.name`	Spark Operator creates a ServiceAccount named `spark-operator-spark` and required RBAC resources in each namespace specified by `spark.jobNamespaces`. If customized, specify the new name when submitting Spark jobs.	`spark-operator-spark`

Step 2: Submit a Spark job

Create a SparkApplication manifest to submit a Spark job.

Create the following manifest and save it as spark-pi.yaml.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default     # The namespace must be in the list of namespaces specified by spark.jobNamespaces.
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
  arguments:
  - "1000"
  sparkVersion: 3.5.4
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark   # If you customized the ServiceAccount name, change the value accordingly.
  executor:
    instances: 1
    cores: 1
    coreLimit: 1200m
    memory: 512m
  restartPolicy:
    type: Never

Submit the Spark job:

kubectl apply -f spark-pi.yaml

Expected output:

sparkapplication.sparkoperator.k8s.io/spark-pi created

Step 3: Monitor the Spark job

Check the status, pods, and logs of the Spark job.

Check the Spark job status:

kubectl get sparkapplication spark-pi

Expected output:

NAME       STATUS      ATTEMPTS   START                  FINISH       AGE
spark-pi   SUBMITTED   1          2024-06-04T03:17:11Z   <no value>   15s

Check the pod status. This filters pods by label sparkoperator.k8s.io/app-name=spark-pi:

kubectl get pod -l sparkoperator.k8s.io/app-name=spark-pi

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
spark-pi-driver                    1/1     Running   0          49s
spark-pi-7272428fc8f5f392-exec-1   1/1     Running   0          13s

After the job completes, the driver deletes all executor pods.

View the Spark job details:

kubectl describe sparkapplication spark-pi

Example output

Output varies based on job status.

Name:         spark-pi
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  sparkoperator.k8s.io/v1beta2
Kind:         SparkApplication
Metadata:
  Creation Timestamp:  2024-06-04T03:16:59Z
  Generation:          1
  Resource Version:    1350200
  UID:                 1a1f9160-5dbb-XXXX-XXXX-be1c1fda4859
Spec:
  Arguments:
    1000
  Driver:
    Core Limit:  1200m
    Cores:       1
    Memory:           512m
    Service Account:  spark
  Executor:
    Core Limit:  1200m
    Cores:       1
    Instances:   1
    Memory:               512m
  Image:                  registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.4
  Image Pull Policy:      IfNotPresent
  Main Application File:  local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
  Main Class:             org.apache.spark.examples.SparkPi
  Mode:                   cluster
  Restart Policy:
    Type:         Never
  Spark Version:  3.5.4
  Type:           Scala
Status:
  Application State:
    State:  COMPLETED
  Driver Info:
    Pod Name:             spark-pi-driver
    Web UI Address:       172.XX.XX.92:0
    Web UI Port:          4040
    Web UI Service Name:  spark-pi-ui-svc
  Execution Attempts:     1
  Executor State:
    spark-pi-26c5XXXXX1408337-exec-1:  COMPLETED
  Last Submission Attempt Time:        2024-06-04T03:17:11Z
  Spark Application Id:                spark-0042dead12XXXXXX43675f09552a946
  Submission Attempts:                 1
  Submission ID:                       117ee161-3951-XXXX-XXXX-e7d24626c877
  Termination Time:                    2024-06-04T03:17:55Z
Events:
  Type    Reason                     Age   From            Message
  ----    ------                     ----  ----            -------
  Normal  SparkApplicationAdded      91s   spark-operator  SparkApplication spark-pi was added, enqueuing it for submission
  Normal  SparkApplicationSubmitted  79s   spark-operator  SparkApplication spark-pi was submitted successfully
  Normal  SparkDriverRunning         61s   spark-operator  Driver spark-pi-driver is running
  Normal  SparkExecutorPending       56s   spark-operator  Executor [spark-pi-26c5XXXXX1408337-exec-1] is pending
  Normal  SparkExecutorRunning       53s   spark-operator  Executor [spark-pi-26c5XXXXX1408337-exec-1] is running
  Normal  SparkDriverCompleted       35s   spark-operator  Driver spark-pi-driver completed
  Normal  SparkApplicationCompleted  35s   spark-operator  SparkApplication spark-pi completed
  Normal  SparkExecutorCompleted     35s   spark-operator  Executor [spark-pi-26c5XXXXX1408337-exec-1] completed

View the last 20 lines of driver pod logs:

kubectl logs --tail=20 spark-pi-driver

Expected output:

24/05/30 10:05:30 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
24/05/30 10:05:30 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 7.942 s
24/05/30 10:05:30 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
24/05/30 10:05:30 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
24/05/30 10:05:30 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 8.043996 s
Pi is roughly 3.1419522314195225
24/05/30 10:05:30 INFO SparkContext: SparkContext is stopping with exitCode 0.
24/05/30 10:05:30 INFO SparkUI: Stopped Spark web UI at http://spark-pi-1e18858fc8f56b14-driver-svc.default.svc:4040
24/05/30 10:05:30 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
24/05/30 10:05:30 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
24/05/30 10:05:30 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
24/05/30 10:05:30 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
24/05/30 10:05:30 INFO MemoryStore: MemoryStore cleared
24/05/30 10:05:30 INFO BlockManager: BlockManager stopped
24/05/30 10:05:30 INFO BlockManagerMaster: BlockManagerMaster stopped
24/05/30 10:05:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/05/30 10:05:30 INFO SparkContext: Successfully stopped SparkContext
24/05/30 10:05:30 INFO ShutdownHookManager: Shutdown hook called
24/05/30 10:05:30 INFO ShutdownHookManager: Deleting directory /var/data/spark-14ed60f1-82cd-4a33-b1b3-9e5d975c5b1e/spark-01120c89-5296-4c83-8a20-0799eef4e0ee
24/05/30 10:05:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-5f98ed73-576a-41be-855d-dabdcf7de189

Step 4: Access the Spark web UI

The web UI is available only while the driver pod is Running.

By default, controller.uiService.enable is true, which creates a Service to expose the web UI for port forwarding. If set to false, no Service is created and you must port-forward from the driver pod directly.

Important

kubectl port-forward is suitable for testing but not recommended for production due to security risks.

Forward the web UI port to your local machine:

Port-forward via Service

kubectl port-forward services/spark-pi-ui-svc 4040

Port-forward via pod

kubectl port-forward pods/spark-pi-driver 4040

Expected output:

Forwarding from 127.0.0.1:4040 -> 4040
Forwarding from [::1]:4040 -> 4040

Open http://127.0.0.1:4040 in your browser.

(Optional) Step 5: Update the Spark job

Update the job manifest to modify Spark job parameters.

Edit spark-pi.yaml. For example, set arguments to 10000 and executor instances to 2.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.4
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.4.jar
  arguments:
  - "10000"
  sparkVersion: 3.5.4
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark # If you customized the ServiceAccount name, change the value accordingly.
  executor:
    instances: 2
    cores: 1
    coreLimit: 1200m
    memory: 512m
  restartPolicy:
    type: Never

Apply the changes:
```
kubectl apply -f spark-pi.yaml
```

Check the job status:

kubectl get sparkapplication spark-pi

The Spark job runs again. Expected output:

NAME       STATUS    ATTEMPTS   START                  FINISH       AGE
spark-pi   RUNNING   1          2024-06-04T03:37:34Z   <no value>   20m

(Optional) Step 6: Delete the Spark job

Delete the Spark job to release its associated resources.

kubectl delete -f spark-pi.yaml

Alternatively:

kubectl delete sparkapplication spark-pi

Container Service for Kubernetes:Run Spark jobs with Spark Operator

Prerequisites

How it works

Procedure overview

Step 1: Deploy the ack-spark-operator component

Step 2: Submit a Spark job

Step 3: Monitor the Spark job

Step 4: Access the Spark web UI

(Optional) Step 5: Update the Spark job

(Optional) Step 6: Delete the Spark job

References