All Products
Search

This Product

    OpenLake:Metadata collection

    Document Center

    OpenLake:Metadata collection

    Last Updated:Sep 16, 2026

    DataWorks Data Map provides the Metadata Collection feature, which allows you to centrally consolidate and manage metadata from various DataWorks data sources. In Data Map, you can view the metadata aggregated from all your sources. This topic describes how to create a crawler that collects and consolidates metadata from various data sources into DataWorks.

    Overview

    Metadata collection runs crawlers to extract technical metadata, data lineage, and partition information from DataWorks data sources such as DLF, MySQL, and CDH Hive across workspaces within the same region. The collected metadata is imported into Data Map for unified management.

    Metadata collection enables you to:

    • Build a unified data view across heterogeneous sources.

    • Discover and search data assets quickly.

    • Trace end-to-end data lineage for impact analysis and troubleshooting.

    • Govern data through classification, access control, quality monitoring, and lifecycle management.

    Billing

    By default, each collection task consumes 0.25 CU multiplied by the task run time, incurring compute unit fees. Each successful collection generates a scheduling instance, incurring scheduling fees.

    Limitations

    • If a data source uses an allowlist for access control, you must configure it beforehand. For more information, see Metadata collection allowlist.

    • The DataWorks deployment and the data source must be in the same region. If you must collect metadata across regions, use a public endpoint when you create the data source. For more information, see Create a data source.

    • Metadata collection is not supported for AnalyticDB for MySQL data sources that have SSL enabled.

    • The new version of metadata collection only supports the new Serverless resource groups. Legacy resource groups are not supported. Make sure that you have created and bound a new Serverless resource group before you use the metadata collection feature.

    Entry point

    1. Log on to the DataWorks console. In the target region, click Data Governance > Data Map in the left-side navigation pane. On the page that appears, click Go to Data Map.

    2. In the left-side navigation pane, click image to go to the metadata collection page.

    Built-in crawlers

    Built-in crawlers are preconfigured and automatically executed (near real-time) by the DataWorks platform. They are used to collect core metadata that is deeply integrated with DataWorks. You do not need to create them. You only need to manage the collection scope.

    Important

    If you cannot find the target table in Data Map, go to My Data > My Tools > Refresh Table Metadata and manually synchronize the relevant tables.

    DLF default crawler

    Important

    To enable real-time collection of DLF metadata, you must grant the Data Reader permission to the service-linked role AliyunServiceRoleForDataworksOnEmr in the DLF console.

    The DLF Default Crawler collects metadata from Data Lake Formation (DLF) under your account.

    1. In the Built-in section on the metadata collection page, find the DLF Default Crawler card and click Details to view basic information.

    2. Switch to the Data Scope tab to view the list of DLF Catalogs that are included in the collection scope and the number of tables they contain.

      By default, all accessible catalogs (including DLF and DLF-Legacy) are collected.

    Custom crawlers

    Custom crawlers allow you to centrally manage metadata across environments and engines.

    • For regular data sources

      You can create custom crawlers for traditional structured or semi-structured data sources such as Hologres, StarRocks, MySQL, Oracle, and CDH Hive. By configuring collection tasks, the system performs deep parsing of physical database and table structures at the source to achieve automated extraction and synchronization of metadata such as column attributes, indexes, and partitions.

    • For metadata-type data sources (catalog)

      For non-DLF-managed, self-declared native lake-format metadata such as Paimon Catalog, Apache Gravitino, and other metadata-type data sources, you can also create crawlers for direct collection.

    Create Custom Crawler

    1. In the custom crawler list section on the metadata collection page, click Create Metadata Crawler.

    2. Select a collection type: On the type selection page, select the target data source type to collect, such as Hologres, StarRocks.

    3. Configure basic settings and resource group:

      • Basic Configurations:

        • Select a workspace: Select the workspace that contains the data source.

        • Select Data Source: Select a previously created target data source from the drop-down list. After you make a selection, the system automatically displays the details of the data source.

        • Name: Name the crawler for easy identification later. By default, the name is the same as the data source name.

      • Resource Group Configuration:

        • Resource Group: Select a Serverless resource group to run the collection task.

        • Test Network Connectivity: This step is critical. Click Test Network Connectivity to make sure that the resource group can access the data source.

          Important

    4. Configure metadata collection:

      • Collection Scope: Define the databases (database/schema) to collect. If the data source is at the database granularity, the database corresponding to the data source is selected by default. You can select additional databases beyond the data source.

        Important

        • Each database can be configured in only one crawler. If a database is grayed out, it is already being collected by another crawler.

        • After you reduce the collection scope, metadata outside the scope is no longer searchable in Data Map.

    5. Configure intelligent enhancement and collection plan:

      • Intelligent enhancement (Beta):

        • AI-generated descriptions: When enabled, the system uses large language model capabilities to automatically generate business descriptions for your tables and columns after metadata is collected, greatly improving metadata readability and usability. After collection is complete, you can view the AI-generated information (such as table descriptions and column descriptions) on the details page of a table object in Data Map.

      • Collection Plan:

        • Trigger Mode: Select manual or periodic.

          • Manual: The crawler runs only when you manually trigger it. This is suitable for one-time or on-demand collection.

          • Periodic: Configure a scheduled task (such as monthly, daily, weekly, or hourly), and the system automatically updates metadata on a periodic basis.

            To configure a minute-level scheduled task, select the hourly collection cycle and select all minute-level granularities to implement a scheduled task that runs every 5 minutes.
            Important

            Only data sources in the production environment support periodic collection.

    6. Save the configuration: Click Save or Save and Run to complete the crawler creation.

    Manage custom crawlers

    After a crawler is created, it appears in the custom crawler list. You can perform the following management operations:

    • List operations: In the list, you can directly Run, Stop, or Delete a crawler. Use the Filter and Search features at the top to quickly locate the target crawler.

      Important

      After you delete a metadata crawler, the metadata objects collected by this crawler in Data Map become invalid. You can no longer search for or view objects and their details from this crawler. Proceed with caution.

    • Batch operations: Select multiple crawlers in the crawler list and use Batch Run or Batch Stop at the bottom of the list to trigger or terminate multiple collection tasks at a time, improving management efficiency.

    • Crawler status: The crawler list displays the current status of each crawler. Common statuses include Not Run, Waiting, Running, Successful, Failed, and Manually Terminated.

      Note

      If the data source associated with a crawler has been disassociated or becomes invalid, the crawler enters the frozen state. A frozen crawler cannot be run and can only be deleted.

    • View details and logs: Click the name of the target crawler to go to its details page.

      • Basic Information: View all configuration items of the crawler.

      • Data Scope: View or Modify Data Scope.

        If no collection has been performed, the table count and last update time are empty.
        The following data sources do not support scope modification: EMR Hive, CDH Hive, Lindorm, ElasticSearch, OTS, MongoDB, and AnalyticDB for Spark in AnalyticDB MySQL.

      • Run Logs: Track the execution history of each collection task. You can view the start time, duration, status, and volume of data collected for each task. When a task fails, clicking View Logs is the key entry point for locating and resolving issues.

    • Manually trigger collection: In the upper-right corner of the details page, click the Collect Metadata button to immediately trigger a collection task. This is useful when you want to see a newly created table in Data Map right away.

    Next steps

    After metadata is successfully collected, you can take full advantage of various Data Map capabilities:

    • Search for your collected tables in Data Map and view their details, column information, partitions, and data preview. For more information, see View table details.

    • Analyze the upstream and downstream lineage of a table to understand the end-to-end data processing flow. For more information, see Data lineage.

    • Add assets to a Data Collection to organize and manage your data from a business perspective. For more information, see Create a Data Collection.

    FAQ