EMR clusters running EMR V3.45.0 or later, or EMR V5.11.0 or later, support JDK 11 for Spark 3 jobs. By default, Spark uses JDK 8. This topic describes how to switch Spark 3 to JDK 11 and how to switch back to JDK 8.
Prerequisites
Before you begin, ensure that your EMR cluster runs EMR V3.45.0 or a later minor version, or EMR V5.11.0 or a later minor version.
Usage notes
EMR Doctor incompatibility: EMR Doctor does not support JDK 11. When switching to JDK 11, remove the EMR Doctor agent from three configuration items: spark.yarn.am.extraJavaOptions, spark.driver.extraJavaOptions, and spark.executor.extraJavaOptions. When switching back to JDK 8, restore those entries.
Switch to JDK 11
On the Configure tab of the Spark3 service page, set the following configuration items. For instructions on modifying configuration items, see Manage configuration items.
spark-env.sh
| Parameter | Value |
|---|---|
spark_java_home |
/usr/lib/jvm/java-11 |
spark-defaults.conf
| Parameter | Value |
|---|---|
spark.yarn.appMasterEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
spark.driverEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
spark.executorEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
spark.yarn.am.extraJavaOptions |
Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
spark.driver.extraJavaOptions |
Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
spark.executor.extraJavaOptions |
Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
If you use Spark Thrift Server, also set the following items in spark-thriftserver.conf:
| Parameter | Value |
|---|---|
spark.yarn.appMasterEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
spark.driverEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
spark.executorEnv.JAVA_HOME |
/usr/lib/jvm/java-11 |
Switch to JDK 8
On the Configure tab of the Spark3 service page, set the following configuration items. For instructions on modifying configuration items, see Manage configuration items.
spark-env.sh
| Parameter | Value |
|---|---|
spark_java_home |
/usr/lib/jvm/java-1.8.0 |
spark-defaults.conf
| Parameter | Value |
|---|---|
spark.yarn.appMasterEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |
spark.driverEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |
spark.executorEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |
spark.yarn.am.extraJavaOptions |
Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
spark.driver.extraJavaOptions |
Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
spark.executor.extraJavaOptions |
Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2 |
If you use Spark Thrift Server, also reset the following items in spark-thriftserver.conf:
| Parameter | Value |
|---|---|
spark.yarn.appMasterEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |
spark.driverEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |
spark.executorEnv.JAVA_HOME |
/usr/lib/jvm/java-1.8.0 |