All Products
Search

This Product

    Container Service for Kubernetes:View Spark jobs with Spark History Server

    Document Center

    Container Service for Kubernetes:View Spark jobs with Spark History Server

    Last Updated:Jun 17, 2026

    Persist Spark job logs and view completed job details, such as stages, tasks, and resource usage, through Spark History Server.

    Prerequisites

    Step 1: Deploy the ack-spark-history-server add-on

    1. Log on to the ACK console. In the left navigation pane, click Marketplace > Marketplace.

    2. On the Marketplace page, click the App Catalog tab, then search for and select ack-spark-history-server.

    3. On the ack-spark-history-server page, click Deploy.

    4. In the Create panel, select a cluster and namespace, then click Next.

    5. On the Parameters page, set the parameters and click OK.

      The following parameters configure log storage, environment variables, and the Service. For all options, see the ConfigMaps tab on the ack-spark-history-server page.

      Note

      When deploying ack-spark-history-server, specify backend storage for logs, such as Object Storage Service (OSS), a persistent volume claim (PVC), or HDFS.

      • (Required) Configure backend storage for logs

        Choose OSS, a PVC, or HDFS as backend storage.

        OSS

        Set the following parameters.

        Important

        Spark job logs are uploaded only after the job completes. You cannot view logs from running jobs in real time.

        Parameter

        Description

        Example

        spark.history.fs.logDirectory

        The log directory URL. Create the directory, such as spark/spark-events, before deploying the add-on. See Managing Directories.

        oss://<Bucket name>/spark/spark-events

        storage.oss.enable

        Enables OSS or OSS-HDFS as backend storage for logs.

        true

        storage.oss.endpoint

        The OSS endpoint.

        oss-cn-beijing-internal.aliyuncs.com

        storage.oss.existingSecret

        (Recommended) The name of an existing Secret with OSS access credentials. See Example of an AccessKey Secret YAML file.

        spark-oss-secret

        storage.oss.createSecret

        If no existing Secret is specified, a new Secret is created automatically to store OSS access credentials.

        Not applicable

        storage.oss.createSecret.accessKeyId

        The AccessKey ID of your Alibaba Cloud account.

        yourAccessKeyID

        storage.oss.createSecret.accessKeySecret

        The AccessKey Secret of your Alibaba Cloud account.

        yourAccessKeySecret

        Example of an AccessKey Secret YAML file

        The Secret must contain the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET fields.

        1. Create a file named spark-oss-secret.yaml:

          apiVersion: v1
kind: Secret
metadata:
  name: spark-oss-secret
  namespace: spark-operator
stringData:
  OSS_ACCESS_KEY_ID: ""       # The AccessKey ID of your Alibaba Cloud account.
  OSS_ACCESS_KEY_SECRET: ""   # The AccessKey Secret of your Alibaba Cloud account.

        2. Create the AccessKey Secret:

          kubectl apply -f spark-oss-secret.yaml

        PVC

        Set the following parameters.

        Parameter

        Description

        Example

        spark.history.fs.logDirectory

        The URL of the log directory.

        file:///mnt/spark/spark-events

        storage.pvc.enable

        Enables a PVC as backend storage for logs.

        true

        storage.pvc.name

        The name of an existing PVC. Ensure the PV and PVC are created and bound in the ack-spark-history-server namespace. See Storage-CSI.

        "<PVC name>"

        storage.pvc.mountPath

        The mount path of the PVC in the container.

        "/mnt"

        HDFS

        Set the following parameter.

        Parameter

        Description

        Example

        spark.history.fs.logDirectory

        The URL of the log directory.

        hdfs://namenode:port/spark/spark-events

      • (Optional) Service type and port

        A Service exposes the History Server web UI by default. Configure the type and port with service.type and service.port.

        Parameter

        Description

        Default

        service.type

        The type of the Service. Valid values:

        • ClusterIP

        • NodePort

        • LoadBalancer

        ClusterIP

        service.port

        The port for accessing the web UI.

        18080

      • (Optional) Environment variables

        Add environment variables in env to configure the History Server.

        Environment variable

        Description

        Default

        SPARK_DAEMON_MEMORY

        The memory allocated to Spark History Server.

        1g

        SPARK_DAEMON_JAVA_OPTS

        The JVM options for Spark History Server.

        ""

        SPARK_DAEMON_CLASSPATH

        The classpath for the Spark History Server.

        ""

        SPARK_PUBLIC_DNS

        The public address of Spark History Server. If unset, the application history uses the internal address, which can cause broken links.

        ""

        SPARK_HISTORY_OPTS

        A set of spark.history.* configuration properties.

        ""

        Add configurations to sparkConf to customize the History Server. Common configurations:

        Property

        Description

        Default

        spark.history.fs.update.interval

        The interval to check for log updates.

        10s

        spark.history.fs.retainedApplications

        The maximum number of application UIs to cache.

        50

        spark.history.ui.port

        The Spark History Server port.

        18080

        spark.history.fs.cleaner.enabled

        Specifies whether to periodically clean up event logs.

        false

        spark.history.fs.cleaner.interval

        The event log cleanup interval when spark.history.fs.cleaner.enabled=true.

        1d

        spark.history.fs.cleaner.maxAge

        Maximum age of event logs before deletion when spark.history.fs.cleaner.enabled=true.

        7d

        spark.history.fs.cleaner.maxNum

        Maximum number of log files to retain when spark.history.fs.cleaner.enabled=true. Excess files are deleted during cleanup.

        Int.MaxValue

        spark.history.fs.driverlog.cleaner.enabled

        Specifies whether to periodically clean up driver event logs.

        spark.history.fs.cleaner.enabled

        spark.history.fs.driverlog.cleaner.interval

        The driver event log cleanup interval when spark.history.fs.driverlog.cleaner.enabled=true.

        spark.history.fs.cleaner.interval

        spark.history.fs.driverlog.cleaner.maxAge

        Maximum age of driver event logs before deletion when spark.history.fs.driverlog.cleaner.enabled=true.

        spark.history.fs.cleaner.maxAge

        spark.history.fs.numReplayThreads

        The number of threads for processing log files.

        25% of the available CPU cores.

        spark.history.store.maxDiskUsage

        The maximum disk space for the application history cache.

        10g

    Step 2: Access the web UI

    By default, the Service type is ClusterIP. Forward a local port to access the web UI. To use an existing SLB instance instead, see Expose an application by using a Service that is associated with an existing SLB instance.

    Important

    kubectl port-forward is for test verification only and is not suitable for production due to security risks.

    1. Configure local port forwarding:

      RELEASE_NAME=spark-history-server
RELEASE_NAMESPACE=spark-operator

SERVICE_NAME=${RELEASE_NAME}-service
SERVICE_PORT=$(kubectl get service ${SERVICE_NAME} --namespace ${RELEASE_NAMESPACE} -o jsonpath="{.spec.ports[0].port}")

echo "Now you can go to http://127.0.0.1:18080 to visit spark history server."
kubectl port-forward --namespace ${RELEASE_NAMESPACE} services/${SERVICE_NAME} 18080:${SERVICE_PORT}

      Expected output:

      Now you can go to http://127.0.0.1:18080 to visit spark history server.
Forwarding from 127.0.0.1:18080 -> 18080
Forwarding from [::1]:18080 -> 18080

    2. Go to http://127.0.0.1:18080 to view the Spark History Server web UI.

      image

    Step 3: Enable event logging

    Enable event logging in your Spark jobs by configuring the following parameters.

    Parameter

    Description

    Example

    spark.eventLog.enabled

    Enables event logging. Valid values:

    • true

    • false

    true

    spark.eventLog.dir

    The storage path for event logs. Examples:

    • oss://<Bucket name>/spark/spark-events (OSS path)

    • hdfs://namenode:port/spark/spark-events (HDFS path)

    • file:///tmp/spark/spark-events (Local path)

    oss://<Bucket name>/spark/spark-events

    Example: Configure OSS for event logging

    The following example configures OSS as backend storage for event logging.

    1. Build a Spark container image

      Community Spark images lack the OSS JAR packages. Build a custom image with the Hadoop OSS SDK JARs shown below, then push it to your repository. See Build an image by using an Enterprise Edition instance. Use dependency JARs matching your Spark version's Hadoop version.

      ARG SPARK_IMAGE=registry-cn-hangzhou.ack.aliyuncs.com/dev/spark:3.5.2

FROM ${SPARK_IMAGE}

# Add dependency for Hadoop Aliyun OSS support
ADD --chmod=644 https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aliyun/3.3.4/hadoop-aliyun-3.3.4.jar ${SPARK_HOME}/jars
ADD --chmod=644 https://repo1.maven.org/maven2/com/aliyun/oss/aliyun-sdk-oss/3.17.4/aliyun-sdk-oss-3.17.4.jar ${SPARK_HOME}/jars
ADD --chmod=644 https://repo1.maven.org/maven2/org/jdom/jdom2/2.0.6.1/jdom2-2.0.6.1.jar ${SPARK_HOME}/jars

    2. Create a Secret

      Create a Secret in your Spark job namespace to store OSS access credentials.

      1. Create the following manifest file spark-oss-secret.yaml:

        apiVersion: v1
kind: Secret
metadata:
  name: spark-oss-secret
  namespace: default
stringData:
  # The AccessKey ID of your Alibaba Cloud account.
  OSS_ACCESS_KEY_ID: ""
  # The AccessKey Secret of your Alibaba Cloud account.
  OSS_ACCESS_KEY_SECRET: ""

      2. Create the Secret:

        kubectl apply -f spark-oss-secret.yaml

        Expected output:

        secret/spark-oss-secret created

    3. Submit the Spark job

      Modify the following parameters in the SparkApplication manifest for your environment:

      Parameter

      Description

      Example

      image

      The custom Spark container image address.

      registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2-oss

      fs.oss.endpoint

      The OSS endpoint.

      oss-cn-beijing-internal.aliyuncs.com

      spark.eventLog.dir

      The log storage path. The path must exist; otherwise the job fails at runtime.

      oss://<Bucket name>/spark/spark-events

      1. Create the following SparkApplication manifest and save it as spark-pi.yaml.

        apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi-oss
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/spark:3.5.2-oss
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.2.jar
  mainClass: org.apache.spark.examples.SparkPi
  sparkVersion: 3.5.2
  hadoopConf:
    fs.AbstractFileSystem.oss.impl: org.apache.hadoop.fs.aliyun.oss.OSS
    fs.oss.impl: org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem
    # The OSS endpoint.
    fs.oss.endpoint: oss-cn-beijing-internal.aliyuncs.com
    fs.oss.credentials.provider: com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
  sparkConf:
    spark.eventLog.enabled: "true"
    # The path where logs are stored.
    spark.eventLog.dir: oss://<Bucket name>/spark/spark-events
  driver:
    cores: 1
    coreLimit: 1200m
    memory: 512m
    serviceAccount: spark-operator-spark
    envFrom:
    - secretRef:
        name: spark-oss-secret
  executor:
    instances: 1
    cores: 1
    coreLimit: 1200m
    memory: 512m
    envFrom:
    - secretRef:
        name: spark-oss-secret
  restartPolicy:
    type: Never

      2. Submit the Spark job. After the job completes, view its history at http://127.0.0.1:18080.

        kubectl apply -f spark-pi.yaml

        Expected output:

        sparkapplication.sparkoperator.k8s.io/spark-pi created