All Products
Search
Document Center

DataWorks:Dedicated Directory

Last Updated:Sep 20, 2026

The dedicated directory is a directory type in the data catalog that is used to directly connect to external metadata services. It is built on the fully managed Apache Gravitino service (metadata service) provided by DataWorks. You do not need to deploy or maintain Gravitino clusters. You can directly access metadata in external metadata services across regions, clouds, and tenants, or in data lakes, manage tables of lake formats and data sources such as Paimon, Iceberg, Hive, and JDBC, and allow compute engines such as Spark to read and write the data in real time. You can also synchronize the metadata to Data Map for unified retrieval and governance. This topic describes how to enable and use the dedicated directory.

Overview

The dedicated directory of the data catalog is built on the fully managed Gravitino service (metadata service) provided by DataWorks. After you enable the metadata service and allocate CU quota when you create a serverless resource group, DataWorks automatically deploys and maintains the Gravitino service in the resource group. You do not need to build or maintain a Gravitino cluster yourself. Compute engines such as Spark can directly connect to the metadata service by using service addresses and access metadata in real time. The metadata is organized by Metalake and catalog.

The data catalog also provides the default directory designed for aggregation and governance. Metadata from various data sources is aggregated into Data Map through metadata collection for unified metadata retrieval and governance, and the metadata is periodically updated based on collection tasks. The two types of directories serve different purposes and do not replace each other. You can use them together. To centrally retrieve and govern the metadata managed in the dedicated directory in Data Map, you can use metadata collection to synchronize the metadata to the default directory. For more information, see the "Synchronize metadata to Data Map" section in this topic.

The dedicated directory provides the following core capabilities:

  • Unified Metalake and catalog management: Create and manage Metalakes in the metadata service on the Dedicated Directory tab of the data catalog. Under each Metalake, create and manage catalogs (such as Paimon, Iceberg, Hive, StarRocks, and JDBC service providers). You can edit, delete, and associate tags and policies with catalogs.

  • Engine read and write access: The service automatically generates the Metadata Service Endpoint and Iceberg Endpoint. After a compute engine such as Spark is connected to the resource group network, the engine can read and write data in the catalogs by using the service addresses.

  • Metadata synchronization and governance: Use metadata collection to synchronize the metadata managed by Gravitino to Data Map for unified metadata retrieval and governance.

Note

If you have a self-managed Gravitino service, connect it to DataWorks for metadata collection. For more information, see Apache Gravitino data source. This topic applies to scenarios in which you use the dedicated directory (fully managed Apache Gravitino service, currently deployed version 1.3.1) of the data catalog. The service provider types that are supported vary with the Gravitino version. The service providers that are available in the console prevail.

Prerequisites

  • A serverless resource group is created or will be created, and the virtual private cloud (VPC) and vSwitch used by the resource group are planned.

  • To access the Gravitino service from a compute engine such as a self-managed Spark cluster, make sure that the network of the engine can communicate with the VPC of the resource group.

  • The required permissions are obtained. To create, edit, or delete Metalakes and Catalogs, the current account must be an Alibaba Cloud account (Tenant Owner), a RAM user that is granted the AliyunDataWorksFullAccess permission, or a member that is granted a tenant-level role such as Tenant Administrator, Data Catalog Administrator, or Metadata Collection Administrator. For more information about how to grant these roles, see Permission control for global services.

Procedure

Step 1: Enable the metadata service

  1. Log on to the DataWorks console and switch to the target region.

  2. In the left-side navigation pane, click Resource Group to go to the resource group list page.

  3. Click Create Resource Group to create a serverless resource group. During the creation, select Metadata Service as the purpose and allocate CUs to the metadata service. The metadata service can be enabled only for subscription serverless resource groups, and the CU quota allocated to the metadata service must be at least 3. For more information about how to create a resource group, see Use serverless resource groups.

  4. After the resource group is created, the system automatically starts the Gravitino service and initializes the metadata store in the resource group. The process takes approximately 5 to 10 minutes. After the service is started, you can use the metadata service on the Dedicated Directory tab of the data catalog.

Note

The metadata service occupies CU resources of the resource group. Plan the CU quota with caution.

Step 2: Create and enter a Metalake

A Metalake is the top-level management unit in the dedicated directory. You can create multiple Metalakes in a resource group, and each Metalake independently manages a set of catalogs. You can divide Metalakes by cloud environment, tenant, or business line. For example, you can create separate Metalakes for production and test tenants, or for data lakes on different clouds, to isolate metadata management and define permission boundaries. Before you access the Metalake management page, you must create or select a Metalake.

  1. Log on to the DataWorks console and switch to the target region. In the left-side navigation pane, click Data Governance > Data Map. On the page that appears, click Go to Data Map.

  2. In the left-side navigation pane of Data Map, click the Data Catalog icon, and then click the Dedicated Directory tab at the top of the page.

  3. If the page indicates that no resource group is available for the metadata service in the current region, the metadata service is not enabled. In this case, click Go to Configuration and enable the metadata service and allocate CU quota for a subscription serverless resource group as described in Step 1. Then, return to the Dedicated Directory tab to continue the subsequent operations.

  4. Click the Metadata Service card that corresponds to the resource group for which the metadata service is enabled to go to the Metalake management page. The first time you access the page, the system guides you through creating a Metalake.

  5. Enter a Metalake name and click OK. The system creates the Metalake and navigates to the Metalake management page. This page provides management entries such as Data Catalogs and Compliance Management.

Step 3: Manage data catalogs

A catalog is the logical unit in the dedicated directory that manages a specific data source. Each catalog corresponds to a data source type, such as Paimon, Iceberg, Hive, StarRocks, or JDBC. On the Data Catalogs tab of the Metalake management page, you can create, edit, and delete catalogs, and associate tags and policies with catalogs.

Create a catalog

  1. On the Metalake management page, make sure that you are on the Data Catalogs tab, and then click + Create Catalog.

  2. In the dialog box that appears, select a service provider. The metadata service supports the following service providers:

    Service provider

    Description

    lakehouse-paimon

    Manages Apache Paimon tables. Suitable for data lakehouse scenarios.

    lakehouse-iceberg

    Manages Apache Iceberg tables. Suitable for open table format scenarios.

    hive

    Manages Apache Hive tables.

    jdbc-mysql / jdbc-postgresql / jdbc-starrocks, etc.

    Manages tables in relational databases. Accesses data sources through JDBC connections.

    lakehouse-generic

    Generic lake format catalog. The actual table format is specified by using parameters. For example, to manage the metadata of external Delta Tables, set format=delta and external=true, and use location to specify the path where the data is stored. This service provider supports metadata management only. You cannot create, modify, or clean up physical Delta data.

  3. Based on the selected service provider, specify the catalog name, connection information (such as the storage path, JDBC URL, and credentials), and other configuration parameters.

  4. Click OK to create the catalog. After the catalog is created, it appears in the data catalog list.

Manage existing catalogs

On the Data Catalogs tab, you can view the name, service provider, description, tags, and policies of all catalogs under the current Metalake. You can perform the following operations on each catalog. To perform operations such as editing and deletion, you must have the required permissions. For more information, see the Prerequisites section of this topic.

  • Edit: Modify the connection information and configuration parameters of the catalog.

  • Delete: Remove a catalog that is no longer in use. Deleting a catalog does not delete the actual data in the underlying data source.

  • Associate tags: Add tags to the catalog for classification and retrieval.

  • Associate policies: Bind access control policies to the catalog for fine-grained permission management.

  • Browse databases and tables: Expand a catalog in the data source tree on the left side to view the databases and table structure details, and edit metadata properties.

Advanced operations

Connect compute engines to read and write data

If you need to use external compute engines such as Spark or EMR to read and write data in catalogs that are managed in the dedicated directory, you must first make sure that the engines can access the Gravitino service addresses, and then configure catalog connections in the engines.

Obtain service addresses

After you enable the metadata service, the system automatically generates the following service addresses for the resource group. You can view the actual addresses by clicking the Metadata Service card on the Dedicated Directory tab of the Data Catalog page.

Service address

Description

Metadata Service Endpoint

The REST API address of the Gravitino metadata service (port 8090). Compute engines or third-party tools use this address to access and manage metadata by calling Gravitino APIs. Example: http://gv-{ID}.{region}.dataworks-metalake.aliyuncs.com:8090

Iceberg Endpoint

The service address of the Iceberg REST Catalog (port 9001). Compute engines use this address to read data from and write data to Iceberg tables by using the Iceberg REST Catalog. Example: http://gv-{ID}.{region}.dataworks-metalake.aliyuncs.com:9001

The domain of a service address is in the format of gv-{ID}.Region.dataworks-metalake.aliyuncs.com. The actual addresses displayed in the console prevail.

Configure engine connections

  1. Deploy the compute engine in the same VPC as the resource group, or use network products to establish connectivity between the engine and the resource group.

  2. Configure a Gravitino catalog or an Iceberg REST Catalog in the engine and specify the service address obtained above. Then, the engine can read data from and write data to the databases and tables in the catalog.

Note

If you use a DataWorks serverless compute engine such as Serverless Spark to access the service, make sure that the VPC bound to the workspace of the engine can communicate with the VPC of the resource group. Otherwise, the Gravitino service is inaccessible.

Synchronize metadata to Data Map

If you need to centrally retrieve and govern the metadata managed by Gravitino in Data Map, you can use metadata collection to synchronize catalog, database, and table metadata to Data Map. If you only manage metadata in the dedicated directory, you can skip this section.

  1. Create an Apache Gravitino data source and set Endpoint to the Metadata Service Endpoint obtained above. For more information, see Apache Gravitino data source.

  2. Create and run a metadata crawler to synchronize the metadata of metalakes, catalogs, databases, and tables managed by Gravitino to Data Map. For more information, see Apache Gravitino metadata collection.

  3. After the synchronization is complete, search for the metadata managed by Gravitino in Data Map and view the details of the databases and tables.

Note

Metadata synchronized to the default directory is not updated in real time. To view the latest metadata, go to the metadata service on the Dedicated Directory tab.