If you use a cluster of E-MapReduce (EMR) V3.45.0 or a later minor version or a cluster of EMR V5.11.0 or a later minor version, you can use Java Development Kit (JDK) 11 in Spark 3 to run a job. This topic describes how to use JDK 11 and how to switch to the default JDK 8 version after you use JDK 11.
Usage notes
EMR Doctor does not support JDK 11. If you want to use JDK 11 to run a Spark job, you must modify the spark.yarn.am.extraJavaOptions, spark.driver.extraJavaOptions, and spark.executor.extraJavaOptions configuration items related to EMR Doctor. For more information, see Use JDK 11.
Use JDK 11
On the Configure tab of the Spark3 service page, modify the following configuration items. For more information about how to modify configuration items, see Manage configuration items.
| Configuration file | Parameter | Value |
| spark-env.sh | spark_java_home | /usr/lib/jvm/java-11 |
| spark-defaults.conf | spark.yarn.appMasterEnv.JAVA_HOME | /usr/lib/jvm/java-11 |
| spark.driverEnv.JAVA_HOME | /usr/lib/jvm/java-11 | |
| spark.executorEnv.JAVA_HOME | /usr/lib/jvm/java-11 | |
| spark.yarn.am.extraJavaOptions | Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. | |
| spark.driver.extraJavaOptions | Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. | |
| spark.executor.extraJavaOptions | Delete -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. |
If you use Spark Thrift Server, you must also modify the following configuration items.
| Configuration file | Parameter | Value |
| spark-thriftserver.conf | spark.yarn.appMasterEnv.JAVA_HOME | /usr/lib/jvm/java-11 |
| spark-thriftserver.conf | spark.driverEnv.JAVA_HOME | /usr/lib/jvm/java-11 |
| spark-thriftserver.conf | spark.executorEnv.JAVA_HOME | /usr/lib/jvm/java-11 |
Switch from JDK 11 to JDK 8
If you want to use the default JDK 8 version in the EMR console, you can roll back the settings of the JDK-related configuration items or reset these configuration items to JDK 8.
On the Configure tab of the Spark3 service page, modify the following configuration items. For more information about how to modify configuration items, see Manage configuration items.
| Configuration file | Parameter | Value |
| spark-env.sh | spark_java_home | /usr/lib/jvm/java-1.8.0 |
| spark-defaults.conf | spark.yarn.appMasterEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 |
| spark.driverEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 | |
| spark.executorEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 | |
| spark.yarn.am.extraJavaOptions | Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. | |
| spark.driver.extraJavaOptions | Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. | |
| spark.executor.extraJavaOptions | Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2. |
If you use Spark Thrift Server, you must also roll back the settings of the following configuration items or reset these configuration items.
| Configuration file | Parameter | Value |
| spark-thriftserver.conf | spark.yarn.appMasterEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 |
| spark-thriftserver.conf | spark.driverEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 |
| spark-thriftserver.conf | spark.executorEnv.JAVA_HOME | /usr/lib/jvm/java-1.8.0 |