View job status

Prerequisites

The Compute Engine service for your Lindorm instance must be activated. For more information, see Activate the service.
Your client's IP address must be in the whitelist for your Lindorm instance. For more information, see Configure a whitelist.

View job status using the Spark web UI

You can use the Spark web UI to view a Spark job's status.

Note

You can view a job's status on the Spark web UI only while it is Running.
To view completed Spark jobs, you must enable the History Server.
For an overview of the Spark web UI, see View Spark job information.

Log on to the Lindorm console.
On the Instances page, click the ID of the target instance.
In the left-side navigation pane, choose Compute Engine > Jobs. This page lists the running Compute Engine jobs.
Click the WebUI Address of the desired Compute Engine job, and log in to the Spark web UI with the Wide Table Engine username and password.

Note
You can find the default username and password for the Wide Table Engine on the Database Connections page, under the Wide Table Engine tab.
In the top navigation bar, click Executors to view details about the running job and its executors.

The Executors page has two sections. The Summary section at the top displays the number of executors and aggregated metrics, categorized as Active, Dead, and Total. The Executors table below lists details for each executor, including its ID, Address, Status, resource usage (Storage Memory, Disk Used, Cores), and task metrics (Active Tasks, Failed Tasks, Complete Tasks, Total Tasks, Task Time (GC Time)).
In the Executors list, click stdout or stderr in the Logs column to view the runtime logs. You can also click Thread Dump to view thread stack information or System Status Dump to view the executor's system status.
Note
- Click stdout to view the standard output log.
- Click stderr to view the standard error log.

View job logs using the HDFS shell

Runtime logs for Compute Engine jobs are automatically stored in the underlying LindormDFS. To review logs from an interrupted job, activate Lindorm and use the HDFS shell.

Important

When you submit a JAR job, initialize the SparkSession object at the beginning of the Main function. This ensures that logs from abnormal runs are also synchronized to LindormDFS.
If your Compute Engine service runs a large number of jobs, the volume of logs might put significant pressure on LindormDFS. To mitigate this, you can set the spark.dfsLog.executor.enabled=false parameter when you start a job. This parameter prevents executor logs from being saved to LindormDFS. The collection of driver logs is not affected.

Log on to the Lindorm console.
On the Instances page, click the ID of the target instance.
In the left-side navigation pane, choose Compute Engine > Jobs, and find the JobId of your job. For example: 562f7c98-2a66-****.
Use the HDFS shell to view the logs. For information about how to configure the HDFS shell, see Connection guide.
The logs for a Compute Engine job are stored in the /ldspark/ldspark-logs/${JobId} directory. The driver logs are stored in the __driver_logs__ subdirectory, and the executor logs are stored in the __executor_logs__/${EXECUTOR_ID} subdirectory. For example, to view the stderr log for the driver, run the following command:
```
$HADOOP_HOME/bin/hadoop fs -cat  /ldspark/ldspark-logs/${JobId}/__driver_logs__/stderr | less
```
Note
You can also mount the LindormDFS directory to an ECS instance by using a FUSE client. For specific steps, see Connect to and use LindormDFS through HDFS FUSE.