All Products
Search

This Product

    E-MapReduce:Use Paimon

    Document Center

    E-MapReduce:Use Paimon

    Last Updated:Jun 02, 2026

    Apache Paimon is a unified data lake format for streaming and batch processing that supports high-throughput writes and low-latency queries. This topic describes how to read and write Paimon tables in EMR Serverless Spark.

    Prerequisites

    A workspace is created. Create a workspace.

    Procedure

    Step 1: Create an SQL session

    1. Go to the Sessions page.

      1. Log on to the EMR console.

      2. In the left-side navigation pane, choose EMR Serverless > Spark.

      3. On the Spark page, click the name of the target workspace.

      4. On the EMR Serverless Spark page, click Sessions in the left-side navigation pane.

    2. On the SQL Sessions page, click Create SQL Session.

    3. On the Create SQL Session page, configure the parameters in the Spark Configuration section and click Create. Manage SQL sessions.

      Spark reads and writes Paimon tables through catalogs. Select a catalog type based on your scenario. Manage data catalogs.

      Data catalog

      When using a data catalog, no session parameters are required. Click Catalogs on the Add Catalog page, then select the data catalog in your Spark SQL development environment.

      Note

      We recommend EMR engine versions esr-4.3.0 or later, esr-3.3.0 or later, or esr-2.7.0 or later.

      Custom catalog

      Data Lake Formation (DLF)

      spark.sql.catalog.<catalogName>                                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                       rest
spark.sql.catalog.<catalogName>.uri                             http://cn-hangzhou-vpc.dlf.aliyuncs.com
spark.sql.catalog.<catalogName>.warehouse                       <catalog_name>
spark.sql.catalog.<catalogName>.token.provider                  dlf
spark.sql.catalog.<catalogName>.dlf.access-key-id               <access_key_id>
spark.sql.catalog.<catalogName>.dlf.access-key-secret           <access_key_secret>

      Parameters:

      Parameter

      Description

      Example value

      spark.sql.catalog.<catalogName>

      The catalog implementation.

      Fixed value: org.apache.paimon.spark.SparkCatalog

      spark.sql.catalog.<catalogName>.metastore

      The metadata storage method. Set to rest to use the DLF REST API.

      Fixed value: rest

      spark.sql.catalog.<catalogName>.uri

      The DLF URI. Format: http://<endpoint>-vpc.dlf.aliyuncs.com.

      http://cn-hangzhou-vpc.dlf.aliyuncs.com

      spark.sql.catalog.<catalogName>.warehouse

      The warehouse path. For DLF, set this to the catalog name.

      <catalog_name>

      spark.sql.catalog.<catalogName>.token.provider

      The authentication provider. Set to dlf for DLF.

      Fixed value: dlf

      spark.sql.catalog.<catalogName>.dlf.access-key-id

      The AccessKey ID of your Alibaba Cloud account or RAM user.

      <access_key_id>

      spark.sql.catalog.<catalogName>.dlf.access-key-secret

      The AccessKey secret of your Alibaba Cloud account or RAM user.

      <access_key_secret>

      DLF-Legacy

      The metadata is stored in DLF 1.0.

      spark.sql.catalog.<catalogName>                          org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                dlf
spark.sql.catalog.<catalogName>.dlf.catalog.id           <catalog_name>
spark.sql.catalog.<catalogName>.dlf.catalog.endpoint     dlf-vpc.cn-hangzhou.aliyuncs.com

      Parameters:

      Parameter

      Description

      Example value

      spark.sql.catalog.<catalogName>

      The catalog implementation.

      Fixed value: org.apache.paimon.spark.SparkCatalog

      spark.sql.catalog.<catalogName>.metastore

      The metadata storage method. Set to dlf to use DLF as the metastore.

      Fixed value: dlf

      spark.sql.catalog.<catalogName>.dlf.catalog.id

      The name of the catalog in DLF.

      <catalog_name>

      spark.sql.catalog.<catalogName>.dlf.catalog.endpoint

      The DLF endpoint. Select a DLF endpoint based on your region.

      dlf-vpc.cn-hangzhou.aliyuncs.com

      Hive Metastore

      The metadata is stored in a specified Hive Metastore.

      spark.sql.catalog.<catalogName>                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore       hive
spark.sql.catalog.<catalogName>.uri             thrift://<yourHMSUri>:<port>

      Parameters:

      Parameter

      Description

      Example value

      spark.sql.catalog.<catalogName>

      The catalog implementation.

      Fixed value: org.apache.paimon.spark.SparkCatalog

      spark.sql.catalog.<catalogName>.metastore

      The metastore type. Set to hive to use a Hive Metastore.

      Fixed value: hive

      spark.sql.catalog.<catalogName>.uri

      The Hive Metastore URI. Format: thrift://<IP address of Hive metastore>:9083.

      <IP address of Hive metastore> is the internal IP address of the Hive Metastore service. To use an external metastore, see Connect to an External Hive Metastore Service.

      thrift://192.168.**.**:9083

      File system

      The metadata is stored in a file system.

      spark.sql.catalog.<catalogName>                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore       filesystem
spark.sql.catalog.<catalogName>.warehouse       oss://<yourBucketName>/warehouse

      Parameters:

      Parameter

      Description

      Example value

      spark.sql.catalog.<catalogName>

      The catalog implementation.

      Fixed value: org.apache.paimon.spark.SparkCatalog

      spark.sql.catalog.<catalogName>.metastore

      The metastore type. Set to filesystem to use a file system as the metastore.

      Fixed value: filesystem

      spark.sql.catalog.<catalogName>.warehouse

      The warehouse path. In this example, <yourBucketName> is the name of an OSS bucket.

      oss://my-bucket/warehouse

      You can configure multiple catalogs simultaneously, such as DLF, DLF 1.0, and Hive. Example:

      # Configure a DLF catalog
spark.sql.catalog.<catalogName>                                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                       rest
spark.sql.catalog.<catalogName>.uri                             http://cn-hangzhou-vpc.dlf.aliyuncs.com
spark.sql.catalog.<catalogName>.warehouse                       <catalog_name>
spark.sql.catalog.<catalogName>.token.provider                  dlf
spark.sql.catalog.<catalogName>.dlf.access-key-id               <access_key_id>
spark.sql.catalog.<catalogName>.dlf.access-key-secret           <access_key_secret>

# Configure a DLF 1.0 catalog
spark.sql.catalog.<catalogName>                                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                       dlf
spark.sql.catalog.<catalogName>.dlf.catalog.id                  <catalog_name>
spark.sql.catalog.<catalogName>.dlf.catalog.endpoint            dlf-vpc.cn-hangzhou.aliyuncs.com


# Configure a hive1 catalog
spark.sql.catalog.<catalogName>                                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                       hive
spark.sql.catalog.<catalogName>.uri                             thrift://<yourHMSUri-1>:<port>

# Configure a hive2 catalog
spark.sql.catalog.<catalogName>                                 org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.<catalogName>.metastore                       hive
spark.sql.catalog.<catalogName>.uri                             thrift://<yourHMSUri-2>:<port>

    Step 2: Read and write Paimon tables

    1. Go to the SQL development page.

      On the EMR Serverless Spark page, click Data Development in the left-side navigation pane.

    2. On the Development tab, click the image icon.

    3. In the Create dialog box, enter a name such as users_task, keep the default type as SparkSQL, and then click OK.

    4. Copy the following code to the new Spark SQL tab (users_task).

      Paimon catalog

      -- Create a database.
CREATE DATABASE IF NOT EXISTS paimon.ss_paimon_db;             

-- Create a Paimon table.
CREATE TABLE paimon.ss_paimon_db.paimon_tbl (id INT, name STRING) USING paimon;

-- Write data to the Paimon table.
INSERT INTO paimon.ss_paimon_db.paimon_tbl VALUES (1, "a"), (2, "b"), (3, "c");

-- Query the write results from the Paimon table.
SELECT * FROM paimon.ss_paimon_db.paimon_tbl ORDER BY id;

-- Delete the database.
DROP DATABASE paimon.ss_paimon_db CASCADE;

    5. Select a database from the database drop-down list and the SQL session you created from the session drop-down list.

    6. Click Run to execute the job. The job returns the following output.

      image

    FAQ

    Why do I receive an error when running a DELETE, UPDATE, or MERGE statement on a table?

    • Symptom: Running a DELETE, UPDATE, or MERGE statement returns an error similar to the following:

      Caused by: org.apache.spark.sql.AnalysisException: Table does not support deletes/updates/merge: <tableName>.
    at org.apache.spark.sql.errors.QueryCompilationErrors$.tableDoesNotSupportError(QueryCompilationErrors.scala:1391)

    • Cause: The table's storage format does not support row-level updates, or a required Spark configuration is missing.

    • Solution:

      1. Check the table type.

        Run the following command to check whether the table is a Paimon table:

        SHOW CREATE TABLE <tableName>;

        If the output contains USING PAIMON, the table is a Paimon table. If the output shows another storage format (such as USING hive), verify that the format supports row-level updates.

      2. Check the Spark configuration.

        If the table is a Paimon table, check the Spark Configuration section and ensure the following configuration is present:

        spark.sql.extensions org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions

        If missing, add it to Spark Configuration.

    Related documentation