This topic describes how to diagnose a Spark on MaxCompute job based on the job log. You can use LogView or the Spark Web UI to check whether a Spark on MaxCompute job is submitted or run.
When you submit a Spark on MaxCompute job by running the
spark-submit script, MaxCompute creates an instance and adds instance information to the LogView
cd $SPARK_HOME bin/spark-submit --master yarn-cluster --class SparkPi /tmp/spark-2.x-demo/target/AliSpark-2.x-quickstart-1.0-SNAPSHOT-shaded.jar
19/01/05 20:36:47 INFO YarnClientImplUtil: logview url: http://logview.odps.aliyun.com/logview/?h=http://service.cn.maxcompute.aliyun.com/api&p=qn_beijing&i=xxx&token=xxx If the job is submitted, the logged information includes but is not limited to the following: 19/01/05 20:37:34 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 11.220.xxx.xxx ApplicationMaster RPC port: 30002 queue: queue start time: 1546691807945 final status: SUCCEEDED tracking URL: http://jobview.odps.aliyun.com/proxyview/jobview/?h=http://service.cn.maxcompute.aliyun-inc.com/api&p=project_name&i=xxx&t=spark&id=application_xxx&metaname=xxx&token=xxx
Diagnose a Spark on MaxCompute job by using LogView
- Open a browser. Then use LogView to view the basic information about your Spark on
MaxCompute job, a cupid-type task.
- On theFuxi Jobstab in the upper pane, find the task namedmaster-0in theTaskNamecolumn. Then, in the lower pane, click theALLtab.
- On the TempRoot tab, find the log you want to view, and click the icon in the StdOut column. Then you can view the log details generated by SparkPi.
Diagnose a Spark on MaxCompute job by using the Spark Web UI
If the log for a Spark on MaxCompute job contains a tracking URL, the job is submitted to the MaxCompute cluster. Both the Spark Web UI and the History Server use this tracking URL.
- Open a browser and enter the tracking URL in the address bar to track your Spark on
- Find the driver you want to view, and then click stdout in the Logs column.