During the test, invalid parameters may lead to exceptions in Apache Spark jobs. This topic describes how to use the Spark UI and ACK Spark History Server to view application status and troubleshoot problems.
Access Spark UI
You can to view the status of each SQL job in Spark UI in real time. The following content describes the procedure:
- Run the kubectl get services command.In this command, tpcds-query-runner-with-alluxio-ui-svc is the Spark UI service.
- Run the following command to access the Spark UI from your on-premises machine:
kubectl port-forward service/tpcds-query-runner-with-alluxio-ui-svc 4040:4040The following figure shows the output of the command:
- In the address bar of a browser, enter localhost:4040 to view jobs in Spark UI.
Access ACK Spark History Server
After a job is completed, you can perform the following operations to view historical data:
- Run the following command to query the ID of the sparkapplication job:
kubectl get sparkapplication tpcds-query-runner-with-alluxio -o yamlThe following figure shows sparkApplicationId.
- Run the following command to query the endpoint of ACK Spark History Server:
kubectl get service ack-spark-history-server
- In the address bar of a browser, enter the IP address (39.XX.XX.XXX) that appears
in the EXTERNAL-IP column in the preceding figure and the port number.Then you can view the historical data of all Spark jobs. By using the sparkApplicationId returned in Step 1, you can find the job that you want to view and perform troubleshooting.