All Products
Search

This Product

    E-MapReduce:Use JDK 11 in Spark 3

    Document Center

    E-MapReduce:Use JDK 11 in Spark 3

    Last Updated:Jul 17, 2026

    Spark 3 in E-MapReduce (EMR) V3.45.0 and later and V5.11.0 and later supports running jobs with Java Development Kit (JDK) 11. You can switch between JDK 11 and the default JDK 8 as needed.

    Usage notes

    EMR Doctor does not support JDK 11. When you run a Spark job with JDK 11, you must also modify the spark.yarn.am.extraJavaOptions, spark.driver.extraJavaOptions, and spark.executor.extraJavaOptions parameters. For more information, see Use JDK 11.

    Use JDK 11

    On the Configure tab of the Spark3 service page, modify the following configuration items. For more information, see Manage configuration items.

    Configuration file

    Parameter

    Value

    spark-env.sh

    spark_java_home

    /usr/lib/jvm/java-11

    spark-defaults.conf

    spark.yarn.appMasterEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    spark.driverEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    spark.executorEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    spark.yarn.am.extraJavaOptions

    Remove -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    spark.driver.extraJavaOptions

    Remove -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    spark.executor.extraJavaOptions

    Remove -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    If you use Spark Thrift Server, you must also modify the following configuration items.

    Configuration file

    Parameter

    Value

    spark-thriftserver.conf

    spark.yarn.appMasterEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    spark-thriftserver.conf

    spark.driverEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    spark-thriftserver.conf

    spark.executorEnv.JAVA_HOME

    /usr/lib/jvm/java-11

    Switch from JDK 11 to JDK 8

    To revert to the default JDK 8, roll back or reset the JDK-related configuration items.

    On the Configure tab of the Spark3 service page, modify the following configuration items. For more information, see Manage configuration items.

    Configuration file

    Parameter

    Value

    spark-env.sh

    spark_java_home

    /usr/lib/jvm/java-1.8.0

    spark-defaults.conf

    spark.yarn.appMasterEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0

    spark.driverEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0

    spark.executorEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0

    spark.yarn.am.extraJavaOptions

    Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    spark.driver.extraJavaOptions

    Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    spark.executor.extraJavaOptions

    Add -noverify -javaagent:/opt/apps/TAIHAODOCTOR/taihaodoctor-current/emr-agent/btrace-agent.jar=libs=spark-3.2.

    If you use Spark Thrift Server, you must also roll back or reset the following configuration items.

    Configuration file

    Parameter

    Value

    spark-thriftserver.conf

    spark.yarn.appMasterEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0

    spark-thriftserver.conf

    spark.driverEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0

    spark-thriftserver.conf

    spark.executorEnv.JAVA_HOME

    /usr/lib/jvm/java-1.8.0