This topic describes how to check whether a job is submitted and executed based on job logs. MaxCompute provides both Logview and Spark Web UI for you to check jobs.
Background information
A job is submitted by using spark-submit. Logs are also generated when you use DataWorks
to execute Spark jobs. Example:
cd $SPARK_HOME
bin/spark-submit --master yarn-cluster --class SparkPi /tmp/spark-2.x-demo/target/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar
After a job is submitted, MaxCompute creates an instance and displays the Logview
and tracking URLs of the instance in a log.
19/01/05 20:36:47 INFO YarnClientImplUtil: logview url: http://logview.odps.aliyun.com/logview/?h=http://service.cn.maxcompute.aliyun.com/api&p=qn_beijing&i=xxx&token=xxx
<The operation succeeded if an output similar to the following result is displayed.>
19/01/05 20:37:34 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 11.220.xxx.xxx
ApplicationMaster RPC port: 30002
queue: queue
start time: 1546691807945
final status: SUCCEEDED
tracking URL: http://jobview.odps.aliyun.com/proxyview/jobview/?h=http://service.cn.maxcompute.aliyun-inc.com/api&p=project_name&i=xxx&t=spark&id=application_xxx&metaname=xxx&token=xxx
Use Logview to check a job
A URL that starts with
logview.odps.aliyun.com
is a Logview URL. Logview is a distributed job tracking tool developed for MaxCompute.
It can be used to:
- Obtain the status of a job.
- Obtain the startup, stop, and scheduling information of each node in a job.
- Obtain the standard input and output logs of each node in a job. We recommend that the Spark output be written into stdout. By default, Spark log4j logs are written into stderr.
- Store execution log data. The data is retained for three to five days. When the local disk is fully occupied, both stdout and stderr are cleared.
Use the Spark Web UI to check a job
The tracking URL in a log indicates that your job is submitted to MaxCompute. You
can use the tracking URL to log on to the Spark Web UI or HistoryServer. The Spark
Web UI can be used to:
- Obtain information of the native Spark Web UI.
- View the information of a running job in real time.
- Transfer events from Spark Driver to HistoryServer after a job is complete. This process
may take one to three minutes. If you open the tracking URL immediately after a job
is complete, error message
Application application_1560240626712_2078769635 not found.
may appear. If this happens, try again later.