Manage data catalogs - E-MapReduce - Alibaba Cloud Documentation Center

A data catalog is the top-level entity for external metadata services such as Data Lake Formation (DLF) or Hive Metastore, and it can contain multiple databases. In EMR Serverless Spark, you can view the databases and tables in a connected data catalog. You can also add existing data catalogs, which is useful for scenarios that require metadata isolation.

Catalog overview

EMR Serverless Spark provides a flexible catalog management system that supports various types of catalogs, such as Paimon, Iceberg, and StarRocks. In addition to the standard data catalog type, the system also supports a built-in catalog and custom catalogs. This section describes the catalog types in EMR Serverless Spark and how to configure and use them.

Catalog type

Supported data sources

How to add

Description

Usage

Data catalog

DLF (formerly DLF 2.5)
DLF-Legacy (formerly DLF 1.0)
Hive Metastore
StarRocks
Hologres

Manually add a catalog on the Catalog page.

After you add a catalog, you must restart any running sessions for the changes to take effect.

On the Development page, select the target catalog in the SQL editor.
Directly run USE <catalogName>; in the SQL editor.
Directly run SELECT * FROM <catalogName>.db.tbl; in the SQL editor.

Custom catalog

Paimon, Iceberg, and others

Edit a session and add the required parameters on the Spark Configuration tab.

This type requires manual parameter configuration and can be extended to support various data sources. For more information, see Use Iceberg and Using Paimon.

-- Switch to the custom catalog
USE <catalogName>;

-- Or reference the table directly
SELECT * FROM <catalogName>.db.tbl;

Add data catalog

Go to the Data Catalogs page.
1. Log on to the EMR console.
2. In the left-side navigation pane, choose EMR Serverless > Spark.
3. On the Spark page, click the name of the target workspace.
4. On the EMR Serverless Spark page, click Catalog in the left-side navigation pane.
  Note
  The Data Catalogs page displays the databases and tables from the Data Lake Formation data catalog that you selected when you created the workspace.
Click Add Catalog.
In the Add Catalog dialog box, select a catalog type, configure the required parameters, and click Add. The following catalog types are supported.

DLF data catalog

DLF Catalog is a service for managing and querying metadata in a data lake. By selecting an existing DLF data catalog or creating a new one, you can quickly access the metadata in your data lake.

To create a new DLF data catalog, click Create Catalog to go to the Data Lake Formation console. For more information, see Manage metadata.

Note

When you use a DLF data catalog, only the following engine versions are supported: esr-4.3.0 or later, esr-3.3.0 or later, and esr-2.7.0 or later.

Hive Metastore (HMS) data catalog

External Hive Metastore is an independent metadata service typically used to manage metadata for Hive tables. You can configure this service to integrate metadata from an external Hive Metastore into your current environment.

Ensure that a network connection exists between EMR Serverless Spark and the VPC where the external Hive Metastore is located.

Parameter	Description
Network Connection	The network connection to the VPC where the external Hive Metastore is located. Select an existing network connection from the drop-down list. For more information, see Step 1: Add a network connection. Note All added data catalogs must share the same network connection. The connection selected for the first catalog is automatically used for all subsequent ones. This network connection becomes the default for all compute resources launched in the workspace and is used to test connectivity to the data catalog. To use a different network connection for a specific data catalog, you must specify it when you submit a job or create a session. A manually specified connection takes precedence.
Metastore service address	The service address of the external Hive Metastore, in the format `thrift://<metastore-host>:<port>`. Where: `<metastore-host>`: The hostname or IP address of the Hive Metastore service. `<port>`: The port number of the Hive Metastore service. The default is `9083`.
Kerberos authentication	If Kerberos authentication is enabled for your external Hive Metastore, specify the keytab file path and the principal name. Kerberos keytab file: The path to the Kerberos keytab file. Kerberos principal: The name of the principal in the keytab file, used to authenticate with the Kerberos service. Note You can use the `klist -kt <keytab_file>` command to view the principal name in the target keytab file.

To add an external Hive Metastore, see Connect to an external Hive Metastore.

StarRocks data catalog

StarRocks is a high-performance analytical database. Adding a StarRocks data catalog registers the metadata of a StarRocks instance in EMR Serverless Spark. After the catalog is added, you can read from and write to StarRocks data directly from your jobs and sessions without configuring connection parameters for each task. Ensure that a network connection exists between EMR Serverless Spark and the VPC where the StarRocks instance is located.

After you add the catalog, you can use a three-part identifier in SQL to read from and write to data in StarRocks. For example: SELECT * FROM <catalogName>.<dbName>.<tableName>;

Note

StarRocks data catalogs are supported only on the following engine versions: esr-4.8.0 or later and esr-5.2.0 or later.

Parameter	Description
Catalog Name	The name for the StarRocks data catalog. This name, which must be unique within the workspace, is used to reference the catalog in SQL.
Normal Network Connection	The network connection to the VPC where the StarRocks instance is located. Select an existing network connection from the drop-down list. For more information, see Step 1: Add a network connection. Note All added data catalogs must share the same network connection. The connection selected for the first catalog is automatically used for all subsequent ones. This network connection becomes the default for all compute resources launched in the workspace and is used to test connectivity to the data catalog. To use a different network connection for a specific data catalog, you must specify it when you submit a job or create a session. A manually specified connection takes precedence.
Endpoint	The frontend (FE) access address for StarRocks, such as `fe--internal.starrocks.aliyuncs.com`.
Query Port	The FE query port for StarRocks. Typically, this is 9030.
Username	The username for accessing StarRocks. Spark jobs use this username for authentication when reading from or writing to StarRocks data.
Password	The password for the specified username.

To learn how to read from and write to StarRocks data in your tasks, see Read from and write to StarRocks.

Hologres data catalog

Hologres is a real-time data warehouse that supports high-throughput real-time writes and real-time analytics on large-scale data. Adding a Hologres data catalog registers the metadata of a Hologres instance in EMR Serverless Spark. After the catalog is added, you can read from and write to Hologres data directly from your jobs and sessions without configuring connection parameters for each task. Ensure that a network connection exists between EMR Serverless Spark and the VPC where the Hologres instance is located.

After you add the catalog, you can use a three-part identifier in SQL to read from and write to data in Hologres. Each catalog is strictly bound to one Hologres database, and cross-database access is not supported. In other words, you cannot use a single catalog to access multiple Hologres databases. The logical structure of the catalog is consistent with Hologres:

Spark concept	Hologres concept	Description
Catalog	Database	For example, `hologres_external_test_db` maps to the `test_db` database in Hologres.
Namespace	Schema	For example, `public` or `test_schema`. The default namespace is `public`. You can run `USE` to switch to a different namespace.
Table	Table	You must explicitly specify the table as `namespace.table_name` (for example, `public.test`), or run `USE namespace` first and then reference the table name directly.

Note

Hologres data catalogs are supported only on the following engine versions: esr-4.9.0 or later.

Parameter	Description
Catalog Name	The name for the Hologres data catalog. This name, which must be unique within the workspace, is used to reference the catalog in SQL.
Normal Network Connection	The network connection to the VPC where the Hologres instance is located. Select an existing network connection from the drop-down list. For more information, see Step 1: Add a network connection. Note All added data catalogs must share the same network connection. The connection selected for the first catalog is automatically used for all subsequent ones. This network connection becomes the default for all compute resources launched in the workspace and is used to test connectivity to the data catalog. To use a different network connection for a specific data catalog, you must specify it when you submit a job or create a session. A manually specified connection takes precedence.
Endpoint	The access address for the Hologres instance. You can find this on the instance details page in the Hologres console.
Query Port	The port for the Hologres instance. Typically, this is 80.
Username	The username for accessing Hologres. Spark jobs use this username for authentication when reading from or writing to Hologres data. For a custom account, use the username in the format `BASIC$<user_name>`. For an Alibaba Cloud account or RAM user, use the AccessKey ID.
Password	The password for the specified username. For a custom account, use the account password. For an Alibaba Cloud account or RAM user, use the AccessKey Secret.

To learn how to read from and write to Hologres data in your tasks, see Read from and write to Hologres.

View databases and tables

On the Catalog page, click a data catalog ID.
All databases in the data catalog are displayed.
In the Actions column, click Table.
All tables in the selected database are displayed.
In the Actions column, click Field.
The schema and column details for the selected table are displayed.