Troubleshoot Spark service exceptions

Spark service logs

When the Spark service encounters an exception, check the service logs for details.

• Service log directories:

  • For E-MapReduce versions later than 3.40.0, 4.10.1, or 5.6.0: /var/log/emr/spark

  • For E-MapReduce 3.40.0, 4.10.1, 5.6.0, and earlier versions: /mnt/disk1/log/spark

• Log files:

  • Spark history server: spark-history-server.log and spark-spark-org.apache.spark.deploy.history.HistoryServer*.out

  • Spark thrift server: spark-thrift-server.log and spark-spark-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2*.out

Spark history server

Disk full due to a large /spark-history directory

• In the EMR console, go to the Spark service configuration page. On the spark-defaults.conf tab, set spark.history.fs.cleaner.enabled to true and restart the Spark history server.

• You can also manually delete the oldest job data from the /spark-history directory in HDFS.

History web UI is inaccessible

1. On the node where the Spark history server is located, run the command ps -ef | grep org.apache.spark.deploy.history.HistoryServer | grep -v grep to check if the Spark history server process exists.

   If the Spark history server process exists, note its process ID (PID).

2. On the same node, runjstat -gcutil [PID] 1000 10. If the values for FGCT and GCT are high and continuously increasing, this indicates insufficient memory. To fix this, go to the EMR console, navigate to the configuration page of the Spark service, select the spark-env.sh tab, increase the value of the spark_history_daemon_memory parameter, and then restart the Spark history server.

Spark thrift server

System event: HealthCheck:Spark.ThriftServer:Failed

1. Check the system event details to see if the message content contains out=-bash: spark-beeline: command not found.

   source    {"nodeName":"emr-header-1","component":"sparkthriftserver","service":"SPARK",}
   message   spark thrift server is failed. code=127, out=-bash: spark-beeline: command not found

   • If yes: Ignore this exception. It is related to a cluster health check.

2. Check if the Spark thrift server process is running with the following command.

   ps -ef | grep org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 | grep -v grep

   If the process does not exist, save the logs, restart the service, and contact technical support.

Connection exception: Thrift saw a transport exception: Could not connect to emr-header-1:10001

1. On the Spark thrift server node, run the following command to check for the process.

   ps -ef | grep org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 | grep -v grep

   If the Spark thrift server process exists, note its process ID (PID).

2. On the same node, run jstat -gcutil [PID] 1000 10, where [PID] is the process ID you obtained in the previous step. If the FGCT and GCT values are high and continuously increasing, the Spark thrift server has insufficient memory. To resolve this, go to the EMR console, navigate to the configuration page of the Spark service, select the spark-env.sh tab, increase the value of the spark_thrift_daemon_memory parameter, and restart the Spark thrift server. Depending on your business requirements, you may need to tune the spark_thrift_daemon_memory value until the service runs normally.

   Alternatively, you can reduce the memory usage of individual Spark SQL queries. On the configuration page of the Spark service, select the spark-thriftserver.conf tab and decrease the value of the spark.driver.maxResultSize parameter. This mitigates the impact of memory-intensive SQL queries on the service.