E-MapReduce:Manage data catalogs

Catalog overview

EMR Serverless Spark provides a flexible catalog management system that supports various types of catalogs, such as Paimon, Iceberg, and StarRocks. In addition to the standard data catalog type, the system also supports a built-in catalog and custom catalogs. This section describes the catalog types in EMR Serverless Spark and how to configure and use them.

-- Switch to the custom catalog
USE <catalogName>;

-- Or reference the table directly
SELECT * FROM <catalogName>.db.tbl;

Add data catalog

1. Go to the Data Catalogs page.

   1. Log on to the EMR console.

   2. In the left-side navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, click the name of the target workspace.

   4. On the EMR Serverless Spark page, click Catalog in the left-side navigation pane.

      Note

      The Data Catalogs page displays the databases and tables from the Data Lake Formation data catalog that you selected when you created the workspace.

2. Click Add Catalog.

3. In the Add Catalog dialog box, select a catalog type, configure the required parameters, and click Add. The following catalog types are supported.

DLF data catalog

DLF Catalog is a service for managing and querying metadata in a data lake. By selecting an existing DLF data catalog or creating a new one, you can quickly access the metadata in your data lake.

To create a new DLF data catalog, click Create Catalog to go to the Data Lake Formation console. For more information, see Manage metadata.

Hive Metastore (HMS) data catalog

External Hive Metastore is an independent metadata service typically used to manage metadata for Hive tables. You can configure this service to integrate metadata from an external Hive Metastore into your current environment.

Ensure that a network connection exists between EMR Serverless Spark and the VPC where the external Hive Metastore is located.

To add an external Hive Metastore, see Connect to an external Hive Metastore.

StarRocks data catalog

StarRocks is a high-performance analytical database. Adding a StarRocks data catalog registers the metadata of a StarRocks instance in EMR Serverless Spark. After the catalog is added, you can read from and write to StarRocks data directly from your jobs and sessions without configuring connection parameters for each task. Ensure that a network connection exists between EMR Serverless Spark and the VPC where the StarRocks instance is located.

After you add the catalog, you can use a three-part identifier in SQL to read from and write to data in StarRocks. For example: SELECT * FROM <catalogName>.<dbName>.<tableName>;

To learn how to read from and write to StarRocks data in your tasks, see Read from and write to StarRocks.

Hologres data catalog

Hologres is a real-time data warehouse that supports high-throughput real-time writes and real-time analytics on large-scale data. Adding a Hologres data catalog registers the metadata of a Hologres instance in EMR Serverless Spark. After the catalog is added, you can read from and write to Hologres data directly from your jobs and sessions without configuring connection parameters for each task. Ensure that a network connection exists between EMR Serverless Spark and the VPC where the Hologres instance is located.

After you add the catalog, you can use a three-part identifier in SQL to read from and write to data in Hologres. Each catalog is strictly bound to one Hologres database, and cross-database access is not supported. In other words, you cannot use a single catalog to access multiple Hologres databases. The logical structure of the catalog is consistent with Hologres:

To learn how to read from and write to Hologres data in your tasks, see Read from and write to Hologres.

View databases and tables

1. On the Catalog page, click a data catalog ID.

   All databases in the data catalog are displayed.

2. In the Actions column, click Table.

   All tables in the selected database are displayed.

3. In the Actions column, click Field.

   The schema and column details for the selected table are displayed.

Use a data catalog in SQL

After you add a data catalog, you can access its databases and tables by catalog name in SQL development and interactive sessions, without specifying connection information in each task.

-- Switch the default catalog
USE <catalogName>;

-- Switch the default catalog and database
USE <catalogName>.<dbName>;

SELECT * FROM <catalogName>.<dbName>.<tableName>;

-- Hologres: the second part is a schema. The default schema is public.
SELECT * FROM <hologresCatalog>.public.<tableName>;

-- You can also switch the catalog and schema first, and then reference the table by name.
USE <hologresCatalog>;
USE public;
SELECT * FROM <tableName>;

SELECT
  d.user_id,
  d.user_name,
  o.order_id,
  o.amount
FROM dlf_catalog.user_db.dim_user AS d
JOIN hologres_catalog.public.orders AS o
  ON d.user_id = o.user_id;

Use a data catalog in PySpark

In PySpark, use three-part identifiers to read from and write to tables in a data catalog.

# Read a table
df = spark.table("<catalogName>.<dbName>.<tableName>")

# Write to a table
(
    df.write
      .mode("append")
      .saveAsTable("<catalogName>.<dbName>.<tableName>")
)

You can also read tables from different data catalogs and then join them.

dim_user = spark.table("dlf_catalog.user_db.dim_user")
orders = spark.table("hologres_catalog.public.orders")

result = orders.join(dim_user, on="user_id", how="inner")
result.show()

Before you write data, make sure that the target data source supports the write mode and that your account has write permissions.

Delete a data catalog

1. On the Catalog page, find the data catalog that you want to delete.

2. Click Delete in the Actions column and confirm the operation as prompted.

After the deletion, running sessions may still use the previous configuration. Restart the sessions and verify that the catalog is no longer accessible.

FAQ