Use Simple Log Service to collect Spark job logs - E-MapReduce

This topic describes how to use Simple Log Service to collect Spark job logs.

Prerequisites

You have created a Spark cluster in the EMR on ACK console. For more information, see Get started with EMR on ACK.
You have activated Simple Log Service (SLS). For more information, see Quick Start: Use Logtail to collect and analyze ECS text logs.

Procedure

Enable the Logtail component for Simple Log Service. For more information, see Collect container logs from ACK clusters.
Note
If the Logtail component for Simple Log Service is already enabled, skip this step and go to Step 2.
Go to the project console in Simple Log Service.
1. Log on to the Container Service for Kubernetes console.
2. On the Clusters page, click the name of the target cluster, or click Details in the Actions column.
3. On the Basic Information page, in the Cluster Resources area, click the link in the Log Service Project row.
  The project console opens.
On the Logstores tab, create two Logstores.
In this example, the two Logstores are named spark-driver-log and spark-executor-log. For more information about how to create a Logstore, see 1. Create a project and a Logstore.

In the spark-driver-log Logstore, do the following:

Create a Logtail configuration. For the data source, select Kubernetes - Standard Output. During the configuration, select an existing Kubernetes machine group.
Select an existing Kubernetes machine group under Data Collection > Logtail Configurations.

Switch to the editor mode and enter the following content.

{
    "inputs": [
        {
            "detail": {
                "IncludeEnv": {
                    "SPARKLOGENV": "spark-driver"
                },
                "Stderr": true,
                "Stdout": true,
                "BeginLineCheckLength": 10,
                "BeginLineRegex": "\\d+/\\d+/\\d+.*"
            },
            "type": "service_docker_stdout"
        }
    ]
}

In the spark-executor-log Logstore, repeat the operations in Step 4 but enter the following content.

{
    "inputs": [
        {
            "detail": {
                "IncludeEnv": {
                    "SPARKLOGENV": "spark-executor"
                },
                "Stderr": true,
                "Stdout": true,
                "BeginLineCheckLength": 10,
                "BeginLineRegex": "\\d+/\\d+/\\d+.*"
            },
            "type": "service_docker_stdout"
        }
    ]
}

Enable indexing for the Logstores. For more information, see Create indexes.
After you complete these steps, you can query the job logs in SLS.