Alibaba Cloud EMR Serverless Spark is a serverless big data processing service built on the Apache Spark engine. Spark applications run on a standard Java Virtual Machine (JVM), and all Spark tasks are executed through Java or Scala code.
Engine versions and meaning of each part in an engine version
The engine version of EMR Serverless Spark follows the format esr-** (Spark **, Scala **).
You can use vector-based technologies and native libraries in the runtime environment provided by the Alibaba Cloud Fusion engine to improve the execution performance of SQL applications. For more information, see Fusion engine.
Item | Description |
esr-** | The engine version of EMR Serverless Spark. Example: esr-2.1. |
Spark ** | The engine version of Apache Spark that is used by EMR Serverless Spark, such as Spark 3.3.1. |
Scala ** | The version of the Scala programming language that is supported by the engine version. Example: Scala 2.12. |
Engine versions and lake table format mapping
4.x series
Engine version | Spark version | Scala version | Delta version | Hudi version | Iceberg version | Paimon version |
esr-4.3.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.1 |
esr-4.2.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 |
esr-4.1.1 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 |
esr-4.1.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 |
esr-4.0.0 | 3.5.2 | 2.12 | 3.2.1 | 0.15 | 1.5.0 | 1.0 |
3.x series
Engine version | Spark version | Scala version | Delta version | Hudi version | Iceberg version | Paimon version |
esr-3.3.0 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.1 |
esr-3.2.0 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 |
esr-3.1.1 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 |
esr-3.1.0 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 |
esr-3.0.1 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 |
esr-3.0.0 | 3.4.3 | 2.12 | 3.0.0 | 0.15 | 1.5.0 | 1.0 |
2.x series
Engine version | Spark version | Scala version | Delta version | Hudi version | Iceberg version | Paimon version |
esr-2.7.0 | 3.3.1 | 2.12 | 2.2.0 | 0.15 | 1.1.0 | 1.1 |
esr-2.6.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 |
esr-2.5.1 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 |
esr-2.5.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 |
esr-2.4.1 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 |
esr-2.4.0 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 1.0 |
esr-2.2.3 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 |
esr-2.2.2 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 |
esr-2.2 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 |
esr-2.1 | 3.3.1 | 2.12 | 2.2.0 | 0.14 | 1.1.0 | 0.9 |