This topic describes how to diagnose a Spark on MaxCompute job based on the job log. You can use LogView or the Spark Web UI to check whether a Spark on MaxCompute job is submitted or run.

Background information

When you submit a Spark on MaxCompute job by running the spark-submit script, MaxCompute creates an instance and adds instance information to the LogView log.

Specifically, you can submit a Spark on MaxCompute job through running the followingspark-submitscript:
cd $SPARK_HOME
bin/spark-submit --master yarn-cluster --class  SparkPi /tmp/spark-2.x-demo/target/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar
After the job is submitted, MaxCompute creates an instance and adds the instance information to the LogView log as follows:
19/01/05 20:36:47 INFO YarnClientImplUtil: logview url: http://logview.odps.aliyun.com/logview/?h=http://service.cn.maxcompute.aliyun.com/api&p=qn_beijing&i=xxx&token=xxx
If the job is submitted, the logged information includes but is not limited to the following:
19/01/05 20:37:34 INFO Client:
   client token: N/A
   diagnostics: N/A
   ApplicationMaster host: 11.220.xxx.xxx
   ApplicationMaster RPC port: 30002
   queue: queue
   start time: 1546691807945
   final status: SUCCEEDED
   tracking URL: http://jobview.odps.aliyun.com/proxyview/jobview/?h=http://service.cn.maxcompute.aliyun-inc.com/api&p=project_name&i=xxx&t=spark&id=application_xxx&metaname=xxx&token=xxx
Note When you run a Spark on MaxCompute task in DataWorks, a similar log is created.

Diagnose a Spark on MaxCompute job by using LogView

  1. Open a browser. Then use LogView to view the basic information about your Spark on MaxCompute job, a cupid-type task.

  2. On theFuxi Jobstab in the upper pane, find the task namedmaster-0in theTaskNamecolumn. Then, in the lower pane, click theALLtab.

  3. On the TempRoot tab, find the log you want to view, and click the icon in the StdOut column. Then you can view the log details generated by SparkPi.

Diagnose a Spark on MaxCompute job by using the Spark Web UI

If the log for a Spark on MaxCompute job contains a tracking URL, the job is submitted to the MaxCompute cluster. Both the Spark Web UI and the History Server use this tracking URL.

  1. Open a browser and enter the tracking URL in the address bar to track your Spark on MaxCompute job.

  2. Find the driver you want to view, and then click stdout in the Logs column.