All Products
Search

This Product

    E-MapReduce:Basic usage

    Document Center

    E-MapReduce:Basic usage

    Last Updated:Mar 26, 2026

    Apache Iceberg on EMR lets you run INSERT, SELECT, UPDATE, and DELETE operations on table-format data stored in OSS, with metadata managed by Data Lake Formation (DLF). This topic walks you through configuring a DLF-backed Iceberg catalog and running basic Spark SQL operations on a Hadoop cluster.

    Prerequisites

    Before you begin, make sure you have:

    Limitations

    Iceberg's Spark SQL extensions are not supported in Spark 2.4. On clusters running EMR V3.38.X or later, use the Spark DataFrame API instead of Spark SQL to interact with Iceberg. This topic covers the Spark SQL approach for EMR V5.3.0 and later.

    Configure the Iceberg catalog

    Before running Spark SQL on Iceberg, configure a catalog that points to DLF as the metadata store. The catalog name and required parameters vary by cluster version. All configurations use the same Spark SQL extension:

    org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions

    The catalog configuration is prefixed with spark.sql.catalog.<catalog_name>, where <catalog_name> is the name assigned to the catalog.

    Version summary:

    Cluster version Default catalog name catalog-impl class
    EMR V5.6.0 or later iceberg org.apache.iceberg.aliyun.dlf.hive.DlfCatalog
    EMR V5.5.X dlf org.apache.iceberg.aliyun.dlf.hive.DlfCatalog
    EMR V5.3.X to V5.4.X dlf_catalog org.apache.iceberg.aliyun.dlf.DlfCatalog

    EMR V5.6.0 or later

    Log on to your cluster via SSH, then run:

    spark-sql --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
 --conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
 --conf spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.aliyun.dlf.hive.DlfCatalog \

    EMR V5.5.X

    Log on to your cluster via SSH, then run:

    spark-sql --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
 --conf spark.sql.catalog.dlf=org.apache.iceberg.spark.SparkCatalog \
 --conf spark.sql.catalog.dlf.catalog-impl=org.apache.iceberg.aliyun.dlf.hive.DlfCatalog \
 --conf spark.sql.catalog.dlf.warehouse=<your-oss-warehouse-path> \
    You can leave spark.sql.catalog.dlf.warehouse blank. If omitted, the default warehouse path is used.

    EMR V5.3.X to V5.4.X

    This version requires an AccessKey pair. Set the ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables before running the command. For instructions, see Configure environment variables in Linux, macOS, and Windows.

    Log on to your cluster via SSH, then run:

    spark-sql --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
 --conf spark.sql.catalog.dlf_catalog=org.apache.iceberg.spark.SparkCatalog \
 --conf spark.sql.catalog.dlf_catalog.catalog-impl=org.apache.iceberg.aliyun.dlf.DlfCatalog \
 --conf spark.sql.catalog.dlf_catalog.io-impl=org.apache.iceberg.hadoop.HadoopFileIO \
 --conf spark.sql.catalog.dlf_catalog.oss.endpoint=<your-oss-endpoint> \
 --conf spark.sql.catalog.dlf_catalog.warehouse=<your-oss-warehouse-path> \
 --conf spark.sql.catalog.dlf_catalog.access.key.id=<ALIBABA_CLOUD_ACCESS_KEY_ID> \
 --conf spark.sql.catalog.dlf_catalog.access.key.secret=<ALIBABA_CLOUD_ACCESS_KEY_SECRET> \
 --conf spark.sql.catalog.dlf_catalog.dlf.catalog-id=<your-catalog-id> \
 --conf spark.sql.catalog.dlf_catalog.dlf.endpoint=<your-dlf-endpoint> \
 --conf spark.sql.catalog.dlf_catalog.dlf.region-id=<your-dlf-region-id>

    Replace the placeholders with your actual values:

    Placeholder Description
    <your-oss-endpoint> OSS endpoint for your region, for example, oss-cn-hangzhou.aliyuncs.com
    <your-oss-warehouse-path> OSS path where Iceberg stores table data, for example, oss://my-bucket/warehouse
    <ALIBABA_CLOUD_ACCESS_KEY_ID> Value of the ALIBABA_CLOUD_ACCESS_KEY_ID environment variable
    <ALIBABA_CLOUD_ACCESS_KEY_SECRET> Value of the ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variable
    <your-catalog-id> DLF catalog ID
    <your-dlf-endpoint> DLF service endpoint
    <your-dlf-region-id> Region ID where DLF is deployed, for example, cn-hangzhou

    Verify the session

    When the Spark SQL CLI is ready, the prompt changes to:

    spark-sql>

    Run basic Iceberg operations

    The examples below use <catalog_name> to refer to the catalog name you configured. For EMR V5.6.0 and later, replace <catalog_name> with iceberg. For other versions, use the catalog name from the Configure the Iceberg catalog section.

    Create a database

    CREATE DATABASE IF NOT EXISTS <catalog_name>.iceberg_db;

    Create a table

    The basic form specifies column names, types, and the Iceberg storage engine:

    CREATE TABLE IF NOT EXISTS <catalog_name>.iceberg_db.sample(
    id BIGINT COMMENT 'unique id',
    data STRING
)
USING iceberg;

    CREATE TABLE also supports the COMMENT, PARTITIONED BY, LOCATION, and TBLPROPERTIES clauses. Use TBLPROPERTIES to set table-level properties, such as the default write format:

    CREATE TABLE IF NOT EXISTS <catalog_name>.iceberg_db.sample(
    id BIGINT COMMENT 'unique id',
    data STRING
)
USING iceberg
TBLPROPERTIES (
    'write.format.default'='parquet'
);

    Insert data

    INSERT INTO <catalog_name>.iceberg_db.sample VALUES (1, 'a'), (2, 'b'), (3, 'c');

    Query data

    SELECT * FROM <catalog_name>.iceberg_db.sample;
SELECT count(1) AS count, data FROM <catalog_name>.iceberg_db.sample GROUP BY data;

    Update data

    UPDATE <catalog_name>.iceberg_db.sample SET data = 'x' WHERE id = 3;

    Delete data

    DELETE FROM <catalog_name>.iceberg_db.sample WHERE id = 3;

    Related topics