EMR Serverless Spark runs on Apache Spark and executes all tasks through Java or Scala code on a standard Java Virtual Machine (JVM). This page describes the engine version format, the component versions bundled with each engine version, and the default Spark configuration overrides that may affect job behavior.
Version number format
Each engine version follows the format esr- (Spark , Scala **).
| Segment |
Meaning |
esr-** |
The EMR Serverless Spark version number. For example, esr-2.1. |
Spark ** |
The Apache Spark version the engine is built on. For example, Spark 3.3.1 means the engine is built on Apache Spark 3.3.1. |
Scala ** |
The Scala version supported by this engine version. For example, Scala 2.12. |
Use the Fusion engine's runtime environment to access vector-based technologies and native libraries, which can improve SQL application execution performance. For more information, see Fusion engine.
Engine versions and lake table format support
The following tables list the component versions bundled with each engine version. A dash (-) means the engine version does not support that component.
5.x series
| Engine version |
Spark version |
Scala version |
Delta Lake version |
Apache Hudi version |
Apache Iceberg version |
Apache Paimon version |
JDK version |
| esr-5.1.0 |
4.0.1 |
2.13 |
- |
- |
- |
1.3 |
JDK 17 |
| esr-5.0.0 |
4.0.1 |
2.13 |
- |
- |
- |
1.3 |
JDK 17 |
4.x series
| Engine version |
Spark version |
Scala version |
Delta Lake version |
Hudi version |
Iceberg version |
Paimon version |
JDK version |
| esr-4.7.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.9.2 |
1.3 |
JDK 17 |
| esr-4.6.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.9.2 |
1.3 |
JDK 17 |
| esr-4.5.1, esr-4.5.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.9.2 |
1.2 |
JDK 17 |
| esr-4.4.1, esr-4.4.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.5.0 |
1.2 |
JDK 8 |
| esr-4.3.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.5.0 |
1.1 |
JDK 8 |
| esr-4.2.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
| esr-4.1.1, esr-4.1.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
| esr-4.0.0 |
3.5.2 |
2.12 |
3.2.1 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
3.x series
| Engine version |
Spark version |
Scala version |
Delta Lake version |
Hudi version |
Iceberg version |
Paimon version |
JDK version |
| esr-3.6.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.3 |
JDK 8 |
| esr-3.5.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.3 |
JDK 8 |
| esr-3.4.1, esr-3.4.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.2 |
JDK 8 |
| esr-3.3.1, esr-3.3.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.1 |
JDK 8 |
| esr-3.2.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
| esr-3.1.1, esr-3.1.0 |
3.4.4 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
| esr-3.0.1, esr-3.0.0 |
3.4.3 |
2.12 |
3.0.0 |
0.15 |
1.5.0 |
1.0 |
JDK 8 |
2.x series
| Engine version |
Spark version |
Scala version |
Delta Lake version |
Hudi version |
Iceberg version |
Paimon version |
JDK version |
| esr-2.9.0 |
3.3.1 |
2.12 |
2.2.0 |
0.15 |
1.5.0 |
1.3 |
JDK 8 |
| esr-2.8.1, esr-2.8.0 |
3.3.1 |
2.12 |
2.2.0 |
0.15 |
1.5.0 |
1.2 |
JDK 8 |
| esr-2.7.1, esr-2.7.0 |
3.3.1 |
2.12 |
2.2.0 |
0.15 |
1.1.0 |
1.1 |
JDK 8 |
| esr-2.6.0 |
3.3.1 |
2.12 |
2.2.0 |
0.14 |
1.1.0 |
1.0 |
JDK 8 |
| esr-2.5.1, esr-2.5.0 |
3.3.1 |
2.12 |
2.2.0 |
0.14 |
1.1.0 |
1.0 |
JDK 8 |
| esr-2.4.1, esr-2.4.0 |
3.3.1 |
2.12 |
2.2.0 |
0.14 |
1.1.0 |
1.0 |
JDK 8 |
| esr-2.2.3, esr-2.2.2, esr-2.2 |
3.3.1 |
2.12 |
2.2.0 |
0.14 |
1.1.0 |
0.9 |
JDK 8 |
| esr-2.1 |
3.3.1 |
2.12 |
2.2.0 |
0.14 |
1.1.0 |
0.9 |
JDK 8 |
Default parameter adjustments
To improve performance and stability, EMR Serverless Spark overrides some Spark default configurations. These overrides apply to all engine versions (esr-2.x through esr-5.x) and may cause job behavior to differ from open-source Spark. Before running jobs, evaluate whether these changes affect your business logic.
| Parameter |
EMR Serverless Spark default |
Open-source Spark default |
Potential impact |
spark.sql.sources.partitionOverwriteMode |
DYNAMIC |
STATIC |
With DYNAMIC, only the partitions included in the output data are overwritten. With STATIC, all existing partitions in the target path are overwritten, even if they contain no new data. Jobs that rely on full partition replacement must account for this difference. |
spark.sql.mapKeyDedupPolicy |
LAST_WIN |
EXCEPTION |
With LAST_WIN, duplicate map keys are silently resolved by keeping the last value. With EXCEPTION, duplicate map keys throw an error. Jobs that previously relied on error-throwing behavior for data validation will not fail as expected. |