Container Service for Kubernetes:Collect Spark job logs with SLS

Prerequisites

• The ack-spark-operator add-on is deployed.

• You have created a Simple Log Service (SLS) Project. For more information, see Manage a Project.

• You have installed the Logtail component.

Overview

This topic walks you through the following steps to configure SLS to manage the system and application logs generated by your Spark jobs.

1. Build a Spark container image that includes the log4j JSON template layout dependency and push the image to your container image registry.

2. Create a ConfigMap to configure Log4j2, set the log level to INFO, and set the log output format to JSONL.

3. Create an AliyunLogConfig resource. Simple Log Service then creates a corresponding Logtail configuration in the specified Logstore to collect logs from Spark jobs submitted by the Spark operator.

4. Create and run a sample Spark job, verify that the pod logs are in JSONL format, and review the definitions of key fields.

5. Log on to the Simple Log Service console to query and analyze Spark job logs within a specific time range.

6. (Optional) Clean up: After you finish testing, delete the Spark jobs and other resources that you created to avoid incurring fees.

Step 1: Build a Spark container image

Create the following Dockerfile (this example uses Spark 3.5.3) and add the required dependency to the Spark classpath. After the build is complete, push the image to your container image registry. To simplify log collection and parsing, we will use the JSONL format for log output.

ARG SPARK_IMAGE=<SPARK_IMAGE>  # Replace <SPARK_IMAGE> with your Spark base image.
FROM ${SPARK_IMAGE}
# Add dependency for log4j-layout-template-json
ADD --chown=spark:spark --chmod=644 https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-layout-template-json/2.24.1/log4j-layout-template-json-2.24.1.jar ${SPARK_HOME}/jars

Step 2: Configure Log4j2 logs

Create a file named spark-log-conf.yaml with the following content. This configuration sets the log level to INFO and the output format to JSONL. It uses the Elastic Common Schema (ECS), a standardized log format, as the log template. For more configuration details, see Collect Log4j logs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-log-conf
  namespace: default
data:
  log4j2.properties: |
    # Set everything to be logged to the console and file
    rootLogger.level = info
    rootLogger.appenderRefs = console, file
    rootLogger.appenderRef.console.ref = STDOUT
    rootLogger.appenderRef.file.ref = FileAppender
    appender.console.name = STDOUT
    appender.console.type = Console
    appender.console.layout.type = JsonTemplateLayout
    appender.console.layout.eventTemplateUri = classpath:EcsLayout.json
    appender.file.name = FileAppender
    appender.file.type = File
    appender.file.fileName = /opt/spark/logs/spark.log
    appender.file.layout.type = JsonTemplateLayout
    appender.file.layout.eventTemplateUri = classpath:EcsLayout.json

Run the following command to create the ConfigMap resource.

kubectl apply -f spark-log-conf.yaml

Expected output:

configmap/spark-log-conf created

Step 3: Create a Logtail configuration

Create an AliyunLogConfig manifest file named aliyun-log-config.yaml with the following content. Replace <SLS_PROJECT> with the name of your SLS Project and <SLS_LOGSTORE> with the name of your Logstore. For more information about configuration options, see Manage Logtail configurations by using AliyunLogConfig.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  name: spark
  namespace: default
spec:
  # (Optional) The name of the destination Project. Default value: k8s-log-<Your_Cluster_ID>.
  project: <SLS_PROJECT>
  # The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates it.
  logstore: <SLS_LOGSTORE>
  # The Logtail configuration.
  logtailConfig:
    # The name of the Logtail configuration.
    configName: spark
    # The type of the data source. `file` indicates text logs.
    inputType: file
    # The configurations for log input.
    inputDetail:
      # The directory where the log files are located.
      logPath: /opt/spark/logs
      # The name of the log file. Wildcard characters are supported.
      filePattern: '*.log'
      # The encoding of the log file.
      fileEncoding: utf8
      # The log type.
      logType: json_log
      localStorage: true
      key:
      - content
      logBeginRegex: .*
      logTimezone: ''
      discardNonUtf8: false
      discardUnmatch: true
      preserve: true
      preserveDepth: 0
      regex: (.*)
      outputType: LogService
      topicFormat: none
      adjustTimezone: false
      enableRawLog: false
      # Collect text logs from containers.
      dockerFile: true
      # Advanced configurations.
      advanced:
        # Preview container metadata.
        collect_containers_flag: true
        # Kubernetes collection configuration.
        k8s:
          # Filter pods by label.
          IncludeK8sLabel:
            sparkoperator.k8s.io/launched-by-spark-operator: "true"
          # Filter containers by name.
          K8sContainerRegex: "^spark-kubernetes-(driver|executor)$"
          # Additional log tag configurations.
          ExternalK8sLabelTag:
            spark-app-name: spark-app-name
            spark-version: spark-version
            spark-role: spark-role
            spark-app-selector: spark-app-selector
            sparkoperator.k8s.io/submission-id: sparkoperator.k8s.io/submission-id
      # Log processing plug-ins.
      plugin:
        processors:
        # Log splitting.
        - type: processor_split_log_string
          detail:
            SplitKey: content
            SplitSep: ''
        # JSON field parsing.
        - type: processor_json
          detail:
            ExpandArray: false
            ExpandConnector: ''
            ExpandDepth: 0
            IgnoreFirstConnector: false
            SourceKey: content
            KeepSource: false
            KeepSourceIfParseError: true
            NoKeyError: false
            UseSourceKeyAsPrefix: false
        # Log timestamp extraction.
        - type: processor_strptime
          detail:
            SourceKey: '@timestamp'
            Format: '%Y-%m-%dT%H:%M:%S.%fZ'
            KeepSource: false
            AdjustUTCOffset: true
            UTCOffset: 0
            AlarmIfFail: false

Run the following command to create the Logtail configuration.

kubectl apply -f aliyun-log-config.yaml

Follow these steps to view the new Logstore and Logtail configuration:

1. Log on to the Simple Log Service console.

2. In the Projects section, click the one you want.

   

3. On the Log Storage > Logstores tab, click the > icon in front of the target logstore, then choose Data Collection > Logtail Configuration.

   On the Logtail configuration page, a list of existing Logtail configurations is displayed. The list includes a configuration named spark, which is associated with one machine group.

4. Click the target Logtail configuration to view its details.

Step 4: Submit a sample Spark job

Create a SparkApplication manifest file named spark-pi.yaml with the following content.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: default
spec:
  type: Scala
  mode: cluster
  image: <SPARK_IMAGE>
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar
  arguments: 
  - "5000"
  sparkVersion: 3.5.3
  sparkConfigMap: spark-log-conf
  driver:
    cores: 1
    memory: 512m
    serviceAccount: spark-operator-spark
  executor:
    instances: 1
    cores: 1
    memory: 4g

Run the following command to submit the job.

kubectl apply -f spark-pi.yaml

After the job is complete, view the last 10 lines of the driver pod log.

kubectl logs  --tail=10 spark-pi-driver  

Expected output:

{"@timestamp":"2024-11-20T11:45:48.487Z","ecs.version":"1.2.0","log.level":"WARN","message":"Kubernetes client has been closed.","process.thread.name":"-937428334-pool-19-thread-1","log.logger":"org.apache.spark.scheduler.cluster.k8s.ExecutorPodsWatchSnapshotSource"}
{"@timestamp":"2024-11-20T11:45:48.585Z","ecs.version":"1.2.0","log.level":"INFO","message":"MapOutputTrackerMasterEndpoint stopped!","process.thread.name":"dispatcher-event-loop-7","log.logger":"org.apache.spark.MapOutputTrackerMasterEndpoint"}
{"@timestamp":"2024-11-20T11:45:48.592Z","ecs.version":"1.2.0","log.level":"INFO","message":"MemoryStore cleared","process.thread.name":"main","log.logger":"org.apache.spark.storage.memory.MemoryStore"}
{"@timestamp":"2024-11-20T11:45:48.592Z","ecs.version":"1.2.0","log.level":"INFO","message":"BlockManager stopped","process.thread.name":"main","log.logger":"org.apache.spark.storage.BlockManager"}
{"@timestamp":"2024-11-20T11:45:48.596Z","ecs.version":"1.2.0","log.level":"INFO","message":"BlockManagerMaster stopped","process.thread.name":"main","log.logger":"org.apache.spark.storage.BlockManagerMaster"}
{"@timestamp":"2024-11-20T11:45:48.598Z","ecs.version":"1.2.0","log.level":"INFO","message":"OutputCommitCoordinator stopped!","process.thread.name":"dispatcher-event-loop-1","log.logger":"org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint"}
{"@timestamp":"2024-11-20T11:45:48.602Z","ecs.version":"1.2.0","log.level":"INFO","message":"Successfully stopped SparkContext","process.thread.name":"main","log.logger":"org.apache.spark.SparkContext"}
{"@timestamp":"2024-11-20T11:45:48.604Z","ecs.version":"1.2.0","log.level":"INFO","message":"Shutdown hook called","process.thread.name":"shutdown-hook-0","log.logger":"org.apache.spark.util.ShutdownHookManager"}
{"@timestamp":"2024-11-20T11:45:48.604Z","ecs.version":"1.2.0","log.level":"INFO","message":"Deleting directory /var/data/spark-f783cf2e-44db-452c-83c9-738f9c894ef9/spark-2caa5814-bd32-431c-a9f9-a32208b34fbb","process.thread.name":"shutdown-hook-0","log.logger":"org.apache.spark.util.ShutdownHookManager"}
{"@timestamp":"2024-11-20T11:45:48.606Z","ecs.version":"1.2.0","log.level":"INFO","message":"Deleting directory /tmp/spark-dacdfd95-f166-4b23-9312-af9052730417","process.thread.name":"shutdown-hook-0","log.logger":"org.apache.spark.util.ShutdownHookManager"}

The logs are printed in JSONL format. The fields are described as follows:

• @timestamp: The time when the log entry was generated.

• ecs.version: The version of Elastic Common Schema (ECS). ECS is a standardized schema for log data.

• log.level: The log level.

• message: The log message.

• process.thread.name: The name of the thread that generated the log entry.

• log.logger: The name of the logger that recorded the log entry.

Step 5: Query and analyze Spark logs

You can refer to the quick start guide for log query and analysis to specify the job's execution time range and confirm that logs were collected successfully.

On the query page of the Simple Log Service console, select the spark Logstore, set a time range, and then run the query. On the Raw Logs tab, you can view the collected Spark application logs. Log entries, such as a WARN-level Unable to load native-hadoop library warning and INFO-level records about the Spark version (3.5.3), operating system, and Java version from the startup phase, confirm that logs are being collected successfully.

(Optional) Step 6: Clean up

If you have finished this tutorial and no longer need the resources, run the following commands to delete them.

Run the following command to delete the Spark job.

kubectl delete -f spark-pi.yaml

Run the following command to delete the Logtail configuration.

kubectl delete -f aliyun-log-config.yaml

Run the following command to delete the Log4j2 log configuration.

kubectl delete -f spark-log-conf.yaml