Collect metadata from DataWorks data sources with crawlers - OpenLake

DataWorks Data Map provides the Metadata Collection feature, which allows you to centrally consolidate and manage metadata from various DataWorks data sources. In Data Map, you can view the metadata aggregated from all your sources. This topic describes how to create a crawler that collects and consolidates metadata from various data sources into DataWorks.

Overview

Metadata collection runs crawlers to extract technical metadata, data lineage, and partition information from DataWorks data sources such as DLF, MySQL, and CDH Hive across workspaces within the same region. The collected metadata is imported into Data Map for unified management.

Metadata collection enables you to:

Build a unified data view across heterogeneous sources.
Discover and search data assets quickly.
Trace end-to-end data lineage for impact analysis and troubleshooting.
Govern data through classification, access control, quality monitoring, and lifecycle management.

Billing

By default, each collection task consumes 0.25 CU multiplied by the task run time, incurring resource group fees. Each successful collection generates a scheduling instance, incurring scheduling instance fees.

Limitations

If a data source uses an allowlist for access control, you must configure it beforehand. For more information, see Metadata collection allowlist.
The DataWorks deployment and the data source must be in the same region. If you must collect metadata across regions, use a public endpoint when you create the data source. For more information, see Data source management.
Metadata collection is not supported for AnalyticDB for MySQL data sources that have SSL enabled.

Entry point

Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.
In the left-side navigation pane, click to open the Metadata Collection page.

Built-in crawlers

The DataWorks platform pre-configures and automatically runs built-in crawlers in near-real time. They are primarily used to collect core metadata from sources that are deeply integrated with DataWorks. You do not need to create them, but you can manage their scope.

Important

If you cannot find a specific table in Data Map, go to My Data > My Tools > Refresh Table Metadata to manually synchronize the table.

DLF default crawler

Important

To enable real-time collection of DLF metadata, you must grant the Data Reader permission to the AliyunServiceRoleForDataworksOnEmr service-linked role in the DLF console.

The DLF Default Crawler collects metadata from Data Lake Formation (DLF) under your account.

On the Metadata Collection page, find the DLF Default Crawler card in the Built-in section and click Details to view its basic information.
Switch to the Data Scope tab to view the list of DLF Catalogs included in the collection scope and the number of tables they contain.

By default, all accessible catalogs, including DLF and DLF-Legacy versions, are collected.

Custom crawlers

Custom crawlers provide unified metadata management capabilities across multiple environments and engines.

For conventional data sources

You can create custom crawlers for traditional structured or semi-structured data sources, such as Hologres, StarRocks, MySQL, Oracle, and CDH Hive. By configuring collection tasks, the system parses the physical structures of databases and tables at the source to automatically extract and synchronize metadata, including column properties, indexes, and partitions.
For catalog-type data sources

For native lake-format metadata that is self-declared and not managed by DLF, such as Paimon Catalog and other metadata-type data sources, you can also create crawlers to collect their metadata directly.

Create Custom Crawler

In the custom crawlers section of the Metadata Collection page, click Create Metadata Collection.
Select collection type: On the type selection page, select the target data source type, such as Hologres or StarRocks.
Configure basic settings and resource group:
- Basic Configurations:
  - Select workspace: Select the workspace where the data source is located.
  - Select Data Source: Select a pre-configured data source from the drop-down list. When you select a data source, the system automatically displays its details.
  - Name: Enter a name for the crawler for future identification. By default, the crawler has the same name as the data source.
- Resource Group Configuration:
  - Resource Group: Select a resource group to run the collection task.
  - Test Network Connectivity : This step is critical. Click Test Network Connectivity to ensure that the resource group can access the data source.
    Important
    
    Confirm whether an allowlist is enabled for the data source. If you need to collect metadata from a data source that has allowlist-based access control enabled, configure allowlist permissions. For more information, see Network connectivity solutions overview and General configurations: Add an IP address or a CIDR block to an allowlist.
    
    If the data source does not have allowlist restrictions, establish a network connection for the data source. For more information, see Resource group operations and network connections.
    
    If the connectivity test returns the error message Backend service call failed: test connectivity failed.not support data type, contact technical support to upgrade the resource group.
Configure metadata collection:
- Collection Scope: Define the databases or schemas to crawl. If the data source is at the database level, the database associated with the data source is selected by default. You can select databases other than the one associated with the data source.
  Important
  - A database can be configured in only one crawler. If a database cannot be selected, it is already being collected by another crawler.
  - If you narrow the collection scope, metadata outside the scope will not be searchable in Data Map.
Configure intelligent enhancement and collection plan:
- Intelligent enhancement configuration (Beta):
  - AI-generated descriptions: If you enable this feature, the system uses large language models (LLMs) to automatically generate business descriptions for your tables and columns after metadata is collected. This greatly improves the readability and usability of your metadata. After the collection is complete, you can view the AI-generated information, such as table and column descriptions, on the table's details page in Data Map.
- Collection Plan:
  - Trigger Mode: Select Manual or Periodic.
    - Manual: The crawler runs only when you manually trigger it. This is suitable for one-time or on-demand collection.
    - Periodic: Configure a schedule, such as hourly, daily, weekly, or monthly, for automatic metadata updates.
      
      To configure a minute-level schedule, set the collection cycle to hourly and select all minute intervals. This allows you to run a scheduled task every 5 minutes.
      
      Important
      Periodic collection is supported only for data sources in a production environment.
Save the configuration: Click Save or Save and Run to complete the crawler creation.

Manage custom crawlers

After a crawler is created, it appears in the custom crawler list. You can perform the following management operations:

List operations: In the list, you can directly perform operations such as Run, Stop, and Delete on crawlers. Use the Filter and Search features at the top of the list to quickly find a specific crawler.

Important
Deleting a crawler removes its collected metadata from Data Map, and you can no longer search for or view these objects and their details. Proceed with caution.
View details and logs: Click a crawler's name to go to its details page.
- Basic Information: View all configuration items of the crawler.
- Data Scope: View or Modify Data Scope.
  
  If a collection has not been run, the table count and last update time are empty.
  
  Scope modification is not supported for the following data sources: EMR Hive, CDH Hive, Lindorm, Elasticsearch, Tablestore, MongoDB, and AnalyticDB for Spark within AnalyticDB for MySQL.
- Run Logs: Track the execution history of each collection task. You can view the task's start time, duration, status, and the amount of data collected. If a task fails, click View Logs to locate and resolve the issue.
Manually run a collection: In the upper-right corner of the details page, click Collect Metadata to immediately trigger a collection task. This is useful when you want a newly created table to appear in Data Map right away.

Next steps

After the metadata is successfully collected, you can use the various features of Data Map:

Search for the tables you have collected in Data Map to view their details, column information, partitions, and data previews. For more information, see View metadata details.
Analyze the upstream and downstream lineage of tables to understand the entire data processing pipeline. For more information, see Data lineage analysis.
Add assets to Data Albums to organize and manage your data from a business perspective. For more information, see Data Albums.

FAQ

Q: Why does a collection task for a database like MySQL time out or fail?

A: Verify that the vSwitch CIDR Block of the resource group is added to the allowlist.