Collect Spark job logs with Simple Log Service - E-MapReduce

You can use Simple Log Service to collect and query logs from Spark jobs running on EMR on ACK.

Prerequisites

You have created a Spark cluster in the EMR on ACK console. For more information, see Get started with EMR on ACK.
You have activated Simple Log Service (SLS). For more information, see Use Logtail to collect and analyze text logs from ECS instances.

Procedure

Enable the Logtail component for Simple Log Service. For more information, see Collect container logs from ACK clusters.

Note
If Logtail is already enabled, skip this step and go to Step 2.
Go to the console for the corresponding Log Service project.
1. Log on to the Container Service for Kubernetes console.
2. On the Clusters page, click the name of the target cluster, or click Details in the Actions column.
3. On the Basic Information page, in the Cluster Resources area, click the link in the Log Service Project row.
  
  The console for the corresponding Log Service project opens.
On the Logstore tab, create two Logstores.

In this example, the two Logstores are named spark-driver-log and spark-executor-log. For more information, see Use Logtail to collect and analyze text logs from ECS instances.

In the spark-driver-log Logstore, perform the following steps.

Create a Logtail configuration. For the data source, select Kubernetes Standard Output. In the configuration wizard, select an existing Kubernetes machine group.
Under Data Collection > Logtail Configurations, select your existing Kubernetes machine group.

Switch to editor mode and enter the following configuration.

{
    "inputs": [
        {
            "detail": {
                "IncludeEnv": {
                    "SPARKLOGENV": "spark-driver"
                },
                "Stderr": true,
                "Stdout": true,
                "BeginLineCheckLength": 10,
                "BeginLineRegex": "\\d+/\\d+/\\d+.*"
            },
            "type": "service_docker_stdout"
        }
    ]
}

In the spark-executor-log Logstore, repeat Step 4 with the following configuration.

{
    "inputs": [
        {
            "detail": {
                "IncludeEnv": {
                    "SPARKLOGENV": "spark-executor"
                },
                "Stderr": true,
                "Stdout": true,
                "BeginLineCheckLength": 10,
                "BeginLineRegex": "\\d+/\\d+/\\d+.*"
            },
            "type": "service_docker_stdout"
        }
    ]
}

Enable indexing for the Logstore. For more information, see Create indexes.

You can now query the job logs in SLS.