E-MapReduce:Use Iceberg

Prerequisites

Before you begin, ensure that you have:

• A workspace in EMR Serverless Spark. See Create a workspace.

Choose a catalog type

Every Iceberg read/write operation in EMR Serverless Spark requires a catalog. Choose a catalog type based on where you store Iceberg metadata.

Step 1: Create a SQL session

1. Log on to the EMR console.

2. In the left-side navigation pane, choose EMR Serverless > Spark.

3. Click the name of the workspace you want to manage.

4. In the left-side navigation pane, choose Operation Center > Sessions.

5. On the SQL Sessions tab, click Create SQL Session.

6. On the Create SQL Session page, go to the Spark Configuration section and add the catalog configuration for your chosen catalog type. Then click Create. For general session settings, see Manage SQL sessions. For data catalog management, see Manage data catalogs.

spark.sql.extensions                                          org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.iceberg_catalog                             org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.iceberg_catalog.catalog-impl                org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.iceberg_catalog.uri                         http://<region-id>-vpc.dlf.aliyuncs.com
spark.sql.catalog.iceberg_catalog.warehouse                   <catalog-name>
spark.sql.catalog.iceberg_catalog.io-impl                     org.apache.iceberg.rest.DlfFileIO
spark.sql.catalog.iceberg_catalog.rest.auth.type              sigv4
spark.sql.catalog.iceberg_catalog.rest.auth.sigv4.delegate-auth-type  none
spark.sql.catalog.iceberg_catalog.rest.signing-region         <region-id>
spark.sql.catalog.iceberg_catalog.rest.signing-name           DlfNext
spark.sql.catalog.iceberg_catalog.rest.access-key-id          <access-key-id>
spark.sql.catalog.iceberg_catalog.rest.secret-access-key      <access-key-secret>

spark.sql.extensions                              org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<catalog-name>                  org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<catalog-name>.catalog-impl     org.apache.iceberg.aliyun.dlf.hive.DlfCatalog
spark.sql.catalog.<catalog-name>.dlf.catalog.id   <catalog-name>

spark.sql.extensions                              org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<catalog-name>                  org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<catalog-name>.catalog-impl     org.apache.iceberg.hive.HiveCatalog
spark.sql.catalog.<catalog-name>.uri              thrift://<hms-ip>:<port>

spark.sql.extensions                              org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
spark.sql.catalog.<catalog-name>                  org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<catalog-name>.type             hadoop
spark.sql.catalog.<catalog-name>.warehouse        oss://<bucket-name>/warehouse

Step 2: Run Iceberg SQL

1. On the EMR Serverless Spark page, click Data Development in the left-side navigation pane.

2. On the Development tab, click the icon.

3. In the New dialog box, enter a name (for example, users_task), leave the type as SparkSQL, and click OK.

4. Paste the following SQL into the new tab.

   Note

   If you omit the database name, tables are created in the default database of the catalog. Create a separate database to organize your tables.

   -- Create a database
   CREATE DATABASE IF NOT EXISTS iceberg_catalog.db;
   
   -- Create a non-partitioned table
   CREATE TABLE iceberg_catalog.db.tbl (
       id BIGINT NOT NULL COMMENT 'unique id',
       data STRING
   )
   USING iceberg;
   
   -- Insert data into the non-partitioned table
   INSERT INTO iceberg_catalog.db.tbl VALUES
   (1, 'Alice'),
   (2, 'Bob'),
   (3, 'Charlie');
   
   -- Query all rows
   SELECT * FROM iceberg_catalog.db.tbl;
   
   -- Query with a filter
   SELECT * FROM iceberg_catalog.db.tbl WHERE id = 2;
   
   -- Update a row
   UPDATE iceberg_catalog.db.tbl SET data = 'David' WHERE id = 3;
   
   -- Confirm the update
   SELECT * FROM iceberg_catalog.db.tbl WHERE id = 3;
   
   -- Delete a row
   DELETE FROM iceberg_catalog.db.tbl WHERE id = 1;
   
   -- Confirm the deletion
   SELECT * FROM iceberg_catalog.db.tbl;
   
   -- Create a partitioned table
   CREATE TABLE iceberg_catalog.db.part_tbl (
       id BIGINT,
       data STRING,
       category STRING,
       ts TIMESTAMP,
       dt DATE
   )
   USING iceberg
   PARTITIONED BY (dt, category);
   
   -- Insert data into the partitioned table
   INSERT INTO iceberg_catalog.db.part_tbl VALUES
     (1 , 'data-01', 'A', timestamp'2026-01-01 10:00:00', date'2026-01-01'),
     (2 , 'data-02', 'A', timestamp'2026-01-01 11:00:00', date'2026-01-01'),
     (3 , 'data-03', 'A', timestamp'2026-01-02 09:30:00', date'2026-01-02'),
     (4 , 'data-04', 'B', timestamp'2026-01-02 12:15:00', date'2026-01-02'),
     (5 , 'data-05', 'B', timestamp'2026-01-03 08:05:00', date'2026-01-03'),
     (6 , 'data-06', 'B', timestamp'2026-01-03 14:20:00', date'2026-01-03'),
     (7 , 'data-07', 'C', timestamp'2026-01-04 16:45:00', date'2026-01-04'),
     (8 , 'data-08', 'C', timestamp'2026-01-04 18:10:00', date'2026-01-04'),
     (9 , 'data-09', 'C', timestamp'2026-01-05 07:55:00', date'2026-01-05'),
     (10, 'data-10', 'A', timestamp'2026-01-05 13:35:00', date'2026-01-05');
   
   -- Query all rows from the partitioned table
   SELECT * FROM iceberg_catalog.db.part_tbl;
   
   -- Query by date partition
   SELECT * FROM iceberg_catalog.db.part_tbl WHERE dt = '2026-01-01';
   
   -- Query by category
   SELECT * FROM iceberg_catalog.db.part_tbl WHERE category = 'A';
   
   -- Query with multiple partition filters
   SELECT * FROM iceberg_catalog.db.part_tbl
   WHERE dt = '2026-01-01'
     AND category = 'A';
   
   -- Aggregate by category
   SELECT category, COUNT(*) AS count
   FROM iceberg_catalog.db.part_tbl
   GROUP BY category;
   
   -- Drop the database (use with caution — the database must be empty first)
   -- DROP DATABASE iceberg_catalog.db;

5. In the session drop-down list, select the SQL session you created, and click Run. Results appear below the editor after the job completes.

   

What's next

• End-to-end task development: Quick start for Spark SQL development — covers the full SQL task development and orchestration workflow.

• Session management: Manage SQL sessions — configure session resources and settings.

• Notebook sessions: Manage notebook sessions — use Iceberg with Notebook instead of Spark SQL.

• Iceberg SQL reference: Apache Iceberg documentation — DDL commands, querying, writing, and table maintenance.