This release adds workspace-level ciphertext management, expands Kyuubi Gateway and Livy Gateway capabilities, and delivers broad engine improvements across fusion acceleration, lakehouse formats, and the Spark framework.
Affected engine versions
The following engine versions include all engine updates described in this release.
| Version | Spark | Scala |
|---|---|---|
| esr-2.7.1 | 3.3.1 | 2.12 |
| esr-2.8.0 | 3.3.1 | 2.12 |
| esr-3.3.1 | 3.4.4 | 2.12 |
| esr-3.4.0 | 3.4.4 | 2.12 |
| esr-4.3.1 | 3.5.2 | 2.12 |
| esr-4.4.0 | 3.5.2 | 2.12 |
| esr-4.5.0 | 3.5.2 | 2.12 |
Platform updates
Workspace-level ciphertext management
Workspaces now support ciphertext management, which lets you store and reference sensitive values—such as AccessKeys and passwords—without exposing plaintext credentials in job configurations.
For details, see Manage sensitive information using ciphertext.
Python version selection for runtime environments
Runtime environments now support specifying a Python version during configuration, giving you control over the Python runtime used by your Spark jobs.
For details, see Manage runtime environments.
Managed file directory mounting in batch jobs
Data development batch jobs now support mounting managed file directories. After mounting, jobs can read from and write to files in the managed file directory directly.
For details, see Managed file directory.
Job history filtering expanded
Job history now supports filtering by resource configuration, Spark configuration, and runtime parameters, making it easier to locate and compare specific job runs.
Kyuubi Gateway improvements
Multiple Kyuubi Gateway instances can now be created within a single workspace.
When associated with a DLF 2.5 Catalog, token creation supports user authorization and performs authentication and authorization for both Kyuubi and DLF 2.5.
For details, see Manage Kyuubi Gateways and Manage data catalogs.
Ranger authorization for Livy Gateway
Livy Gateway now supports configuring and enabling Ranger authorization.
For details, see Configure and enable Ranger authorization for Livy Gateway.
R language support in Notebook
Notebook now supports the R language in addition to existing supported languages.
Engine updates
Fusion acceleration
Sort operator optimization.
Spill optimization.
JSON Datasource optimization.
Project merge optimization.
Optimization for text deduplication scenarios in large models.
Spark SQL now supports the OSS restore syntax.
Upgraded Jindo SDK to 6.10.0.
Fine-grained UDF fallback optimization.
Optimized
regexp_replace.Integrated DuckDB into Notebook.
Lakehouse formats
[DLF 2.5] Added full support for DLF 2.5 PaimonCatalog and IcebergCatalog.
[Paimon] Optimized read and write operations.
[Paimon] Added support for federated queries and writes across multiple catalogs.
[Paimon] Optimized the file open cost configuration.
[Iceberg] Added support for multiple catalogs.
[Lance] Added support for DLF Lance Table.
Spark framework
Upgraded esr-4.5.0 to Java 17.
Optimized insert overwrite for non-partitioned tables.
Improved compatibility between Spark and Hive.
Optimized the MC connector.
Jobs now fail automatically after consecutive executor failures.
Celeborn
Shuffle write skew optimization.
Stability optimization.
Performance optimization.