Use Spark History Server to view Spark job information - Container Service for Kubernetes

A running Spark application provides a web UI to visualize job information, such as stage details, task details, and memory usage. To view job execution details after a job is complete, you must persist the Spark job logs to a backend storage system. You can then use the ack-spark-history-server component to parse the logs and display them in a web UI. This topic describes how to use Spark History Server to view the status of Spark jobs in an ACK cluster.

Prerequisites

The ack-spark-operator component is deployed. For more information, see Deploy the ack-spark-operator component.
A kubectl client is connected to the ACK cluster. For more information, see Get a cluster kubeconfig and connect to the cluster using kubectl.

Step 1: Deploy the ack-spark-history-server component

Log on to the Container Service for Kubernetes (ACK) console. In the navigation pane on the left, choose Marketplace > App Marketplace.
On the App Marketplace page, click the App Catalog tab. Search for and select ack-spark-history-server.
On the ack-spark-history-server page, click Deploy.
In the Create panel, select a cluster and namespace, and then click Next.

On the Parameter Settings page, set the parameters and click OK.

The following sections describe the parameters for log storage, environment variables, and Service configuration. You can customize the parameter settings as needed. You can view the descriptions of all configuration items on the Parameters tab of the ack-spark-history-server page.

Note

When you deploy the ack-spark-history-server component, you must specify a backend log storage system, such as OSS, a persistent volume claim (PVC), or HDFS.

(Required) Configure a backend log storage system

When you configure a backend log storage system, you can choose OSS, a PVC, or HDFS. The following sections describe the configurations for each storage backend.

Use OSS as a storage backend

If you use OSS as the log storage backend, set the following parameters.

Important

Spark jobs write logs and upload them only after the job is complete. Therefore, you cannot view the logs of a running job in real time. You can view only the logs of completed jobs.

Parameter	Description	Example
`spark.history.fs.logDirectory`	The URL of the log directory. Make sure that the log directory, such as `spark/spark-events`, is created before you deploy the component. For more information about how to create a directory, see Manage directories.	`oss://<Bucket name>/spark/spark-events`
`storage.oss.enable`	Enables OSS and OSS-HDFS as the log storage backend.	`true`
`storage.oss.endpoint`	The OSS endpoint.	`oss-cn-beijing-internal.aliyuncs.com`
`storage.oss.existingSecret`	(Recommended) The name of an existing Secret that contains the credentials to read from OSS. For more information about how to create a Secret YAML file for OSS access credentials, see Sample Secret YAML for OSS access credentials.	`spark-oss-secret`
`storage.oss.createSecret`	If you do not specify an existing Secret, a Secret is automatically created to store the OSS access credentials.	Not applicable.
`storage.oss.createSecret.accessKeyId`	Your AccessKey ID.	yourAccessKeyID
`storage.oss.createSecret.accessKeySecret`	Your AccessKey secret.	yourAccessKeySecret

Sample Secret YAML for OSS access credentials

Make sure that the Secret contains the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET fields.

Create a file named spark-oss-secret.yaml with the following content.

apiVersion: v1
kind: Secret
metadata:
  name: spark-oss-secret
  namespace: spark-operator
stringData:
  OSS_ACCESS_KEY_ID: ""       # Your AccessKey ID.
  OSS_ACCESS_KEY_SECRET: ""   # Your AccessKey secret.

Run the following command to create the Secret for the OSS access credential.
```
kubectl apply -f spark-oss-secret.yaml
```

Use a PVC as a storage backend

If you use a PVC as the log storage backend, set the following parameters.

Parameter	Description	Example
`spark.history.fs.logDirectory`	The URL of the log directory.	`file:///mnt/spark/spark-events`
`storage.pvc.enable`	Enables a PVC as the log storage backend.	`true`
`storage.pvc.name`	The name of an existing PVC. Make sure that a persistent volume (PV) and a PVC are created in the namespace required by `ack-spark-history-server` and are successfully bound. For more information about how to create and bind a PVC, see Storage - CSI.	`"<PVC name>"`
`storage.pvc.mountPath`	The mount path of the PVC in the container.	`"/mnt"`

Use HDFS as a storage backend

If you use HDFS as the log storage backend, set the following parameter.

Parameter	Description	Example
`spark.history.fs.logDirectory`	The URL of the log directory.	`hdfs://namenode:port/spark/spark-events`

(Optional) Configure the Service type and port
By default, a Service resource is created to expose the History Server web UI. You can configure the Service type and port number using the service.type and service.port parameters.
Parameter
Description
Default value
service.type
The Service type. Valid values:
ClusterIP
NodePort
LoadBalancer
ClusterIP
service.port
The port for accessing the web UI.
18080

(Optional) Configure environment variables

You can add environment variables in the env parameter to configure the History Server.

Environment variable	Description	Default value
`SPARK_DAEMON_MEMORY`	The amount of memory allocated to the History Server.	`1g`
`SPARK_DAEMON_JAVA_OPTS`	The JVM configuration item for the History Server.	`""`
`SPARK_DAEMON_CLASSPATH`	The class path of the History Server.	`""`
`SPARK_PUBLIC_DNS`	The public address of the History Server. If the public address of the history server is not set, the application history uses the internal address by default, which may cause links to fail.	`""`
`SPARK_HISTORY_OPTS`	A set of `spark.history.*` configuration items.	`""`

You can add configurations in the sparkConf parameter to set up the History Server. The following table describes common configuration items.

Property name	Description	Default value
`spark.history.fs.update.interval`	The interval at which to check for log updates.	`10s`
`spark.history.fs.retainedApplications`	The maximum number of applications for which UI data is cached.	`50`
`spark.history.ui.port`	The port number of the History Server.	`18080`
`spark.history.fs.cleaner.enabled`	Specifies whether to periodically clean up event logs.	`false`
`spark.history.fs.cleaner.interval`	When `spark.history.fs.cleaner.enabled=true`, this parameter specifies the interval at which to clean up event logs.	`1d`
`spark.history.fs.cleaner.maxAge`	When `spark.history.fs.cleaner.enabled=true`, logs that have a time to live (TTL) longer than this threshold are deleted during cleanup.	`7d`
`spark.history.fs.cleaner.maxNum`	When `spark.history.fs.cleaner.enabled=true`, this parameter sets the maximum number of log files to retain. Log files that exceed this threshold are deleted when the cleanup is triggered.	`Int.MaxValue`
`spark.history.fs.driverlog.cleaner.enabled`	Specifies whether to periodically clean up driver event logs.	`spark.history.fs.cleaner.enabled`
`spark.history.fs.driverlog.cleaner.interval`	When `spark.history.fs.driverlog.cleaner.enabled=true`, this parameter specifies the interval at which to clean up driver event logs.	`spark.history.fs.cleaner.interval`
`spark.history.fs.driverlog.cleaner.maxAge`	When `spark.history.fs.driverlog.cleaner.enabled=true`, driver event logs that have a TTL longer than this threshold are deleted during cleanup.	`spark.history.fs.cleaner.maxAge`
`spark.history.fs.numReplayThreads`	The number of threads created to process log files.	25% of the available CPU cores.
`spark.history.store.maxDiskUsage`	The maximum disk size that can be used for application history caching.	`10g`

Step 2: Access the Spark History Server web UI

By default, the Service type is ClusterIP. To access the Spark History Server web UI, you can forward its Service to local port 18080. You can configure port forwarding as described in the following steps. To use an existing SLB instance for access, see Expose an application using a Service that is associated with an existing SLB instance.

Important

Port forwarding that is established using the kubectl port-forward command is suitable only for quick verification in staging environments and is not recommended for production environments due to security risks.

Run the following command to configure local port forwarding.

RELEASE_NAME=spark-history-server
RELEASE_NAMESPACE=spark-operator

SERVICE_NAME=${RELEASE_NAME}-service
SERVICE_PORT=$(kubectl get service ${SERVICE_NAME} --namespace ${RELEASE_NAMESPACE} -o jsonpath="{.spec.ports[0].port}")

echo "Now you can go to http://127.0.0.1:18080 to visit spark history server."
kubectl port-forward --namespace ${RELEASE_NAMESPACE} services/${SERVICE_NAME} 18080:${SERVICE_PORT}

Expected output:

Now you can go to http://127.0.0.1:18080 to visit spark history server.
Forwarding from 127.0.0.1:18080 -> 18080
Forwarding from [::1]:18080 -> 18080

In a browser, open http://127.0.0.1:18080 to view the Spark History Server web UI.

Step 3: Enable logging in a Spark job

After you deploy the ack-spark-history-server component, you must enable event logging in your Spark job. You can configure the following two parameters to enable and store event logs for the Spark job.

Parameter

Description

Sample value

spark.eventLog.enabled

Enables event logging. Valid values:

true
false

true

spark.eventLog.dir

The path where event logs are stored. Valid values:

oss://<Bucket name>/spark/spark-events (OSS path)
hdfs://namenode:port/spark/spark-events (HDFS path)
file:///tmp/spark/spark-events (Local path)

oss://<Bucket name>/spark/spark-events

Example scenario: Configure OSS logging in a Spark job

The following steps describe how to configure logging with OSS as the storage backend.

Build a Spark container image
The community Spark container images do not include the JAR packages that are required to access OSS. You must build your own Spark container image and add the JAR packages from the Hadoop OSS SDK to the image. The following code provides a sample Dockerfile. For more information about how to build an image using Container Registry, see Use an Enterprise Edition instance to build an image. Build an image based on the Dockerfile and push it to your image repository. Make sure that you adjust the dependency JAR packages to match the Hadoop version of Spark.
```
ARG SPARK_IMAGE=registry-cn-hangzhou.ack.aliyuncs.com/dev/spark:3.5.2

FROM ${SPARK_IMAGE}

# Add dependency for Hadoop Aliyun OSS support
ADD --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars
```
Create a Secret
In the namespace of the Spark job, create a Secret to store the OSS access credentials.
1. Create a Secret manifest file named spark-oss-secret.yaml to store the OSS access credentials.
```
apiVersion: v1
kind: Secret
metadata:
  name: spark-oss-secret
  namespace: default
stringData:
  # Your AccessKey ID
  OSS_ACCESS_KEY_ID: ""
  # Your AccessKey secret
  OSS_ACCESS_KEY_SECRET: ""
```
2. Run the following command to create the Secret resource.
```
kubectl apply -f spark-oss-secret.yaml
```
  Expected output:
```
secret/spark-oss-secret created
```

Submit a Spark job

The following code provides a sample SparkApplication with event logging enabled. Modify the following parameters as needed:

Parameter	Description	Example
`image`	The registry address of the built Spark container image.	`registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2-oss`
`fs.oss.endpoint`	The Alibaba Cloud OSS endpoint.	`oss-cn-beijing-internal.aliyuncs.com`
`spark.eventLog.dir`	The path where logs are stored. Make sure that the specified log path is created in advance. Otherwise, an error occurs when the job runs.	`oss://<Bucket name>/spark/spark-events`

Create the following SparkApplication manifest file and save it as spark-pi.yaml.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-oss
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2-oss
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  mainClass: org.apache.spark.examples.SparkPi
  sparkVersion: 3.5.2
  hadoopConf:
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    # OSS endpoint
    fs.oss.endpoint: oss-cn-beijing-internal.aliyuncs.com
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  sparkConf:
    spark.eventLog.enabled: "true"
    # Log storage path
    spark.eventLog.dir: oss://<Bucket name>/spark/spark-events
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    envFrom:
    - secretRef:
        name: spark-oss-secret
  executor:
    instances: 1
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-oss-secret
  restartPolicy:
    type: Never

Run the following command to submit the Spark job. After the image is built and the Secret is created, you can open http://127.0.0.1:18080 in a browser to view the Spark job status.
```
kubectl apply -f spark-pi.yaml
```
Expected output:
```
sparkapplication.sparkoperator.k8s.io/spark-pi created
```