All Products
Search

This Product

    E-MapReduce:What is EMR Serverless Spark

    Document Center

    E-MapReduce:What is EMR Serverless Spark

    Last Updated:May 27, 2026

    EMR Serverless Spark is a fully managed lakehouse engine for data and AI workloads. It is 100% compatible with open-source Spark -- run existing jobs with spark-submit and spark-sql without code changes.

    Resources scale within seconds at single-core granularity and release immediately after each job. You pay only for actual consumption.

    Use cases

    • Data warehousing and BI analytics: Run SQL queries and build reports through the built-in SQL editor. Compatible with traditional data warehouse workflows.

    • ETL and data engineering: Orchestrate batch processing, stream computing, and data transformation pipelines in a single workflow.

    • Machine learning and data science: Develop and train models interactively in the built-in Notebook with Python environment management and SparkML.

    • Lakehouse analytics: Query and manage data stored in open lakehouse formats such as Apache Paimon, Apache Iceberg, Delta Lake, and Apache Hudi.

    Architecture

    image

    The architecture has four layers:

    Application scenario layer

    The SQL editor handles queries and reports. The Notebook provides interactive Python and ML development. Both share a unified platform for seamless transitions between analytics and model training.

    Platform capability layer

    Workflow orchestration schedules batch, streaming, and AI jobs in one pipeline. Resource Access Management (RAM) provides fine-grained access control. The Notebook, Apache Kyuubi, and Apache Livy offer flexible programming interfaces and task submission.

    Core engine layer

    Two built-in engines accelerate performance:

    Engine

    Description

    Fusion Engine (Spark Native Engine)

    A C++-based vectorized SQL engine using SIMD instructions. Delivers 300% better performance than open-source Spark by improving CPU utilization and reducing memory overhead versus JVM execution.

    Celeborn (Remote Shuffle Service)

    An enterprise-grade remote shuffle service that handles petabyte-scale shuffle data with multi-tenant isolation. Eliminates large disks on compute nodes, fully leverages dynamic resource scaling, and reduces computing costs by up to 30%.

    Lakehouse storage layer

    Built on Apache Paimon and Apache Iceberg, this layer combines data lake flexibility with warehouse capabilities: ACID transactions, efficient upserts, and data lineage tracking.

    Elastic scaling and cost efficiency

    • Compute-storage separation: Compute scales within seconds at single-core granularity. Storage uses pay-as-you-go pricing.

    • Task-level metering: Metered per task or queue, with built-in cost estimation.

    • HDFS-compatible cloud storage: Integrates with OSS-HDFS for smooth on-premises migration. Uses Data Lake Formation (DLF) for unified metadata management and access control.

    Ecosystem compatibility

    • Spark compatibility: 100% compatible with open-source Spark. Use spark-submit and spark-sql without modification.

    • Lakehouse formats: Supports Apache Paimon, Apache Iceberg, Delta Lake, and Apache Hudi.

    • Scheduling integration: Works with Apache Airflow and Apache DolphinScheduler.

    • Security: Connects to Kerberos or LDAP for authentication. Uses Apache Ranger for data authorization.

    • Machine learning: Built-in SparkML and Notebook with Python library lifecycle management.

    Development platform

    • End-to-end workflow: Develop, debug, publish, and schedule tasks in one platform.

    • Version management: Full release history with source and configuration diffs. All changes are traceable.

    • Environment isolation: Development and production environments are strictly separated.

    Serverless operations

    • No infrastructure setup: Start immediately with no cluster management or configuration.

    • Automatic resource management: Pods start dynamically per task requirements and release immediately after computation.

    • Pay for what you use: Billed by actual resource consumption.

    Billing

    EMR Serverless Spark supports two billing methods:

    Billing method

    Description

    Subscription

    Purchase resources for a specific period. Pay before you use the resources.

    Pay-as-you-go

    Activate and release resources as needed. Pay after you use the resources.

    Get started