DataWorks:Collect Metadata

MaxCompute Default Crawler

This crawler collects metadata from the MaxCompute projects under your account. You can go to the details page to select the projects for collection using the Modify Data Scope option and set metadata visibility within the tenant using the Permission Configurations option.

1. On the collect metadata page, in the Built-in section, find the MaxCompute Default Crawler card and click Details.

2. The MaxCompute Default Crawler details page contains the Basic Information and Data Scope tabs.

   • Basic Information: Displays the basic properties of the crawler, such as the collection type and method. This information is read-only.

   • Data Scope: Manage the MaxCompute projects from which the crawler collects metadata.

3. Modify collection scope:

   1. Switch to the Data Scope tab and click the Modify Data Scope button.

   2. In the dialog box that appears, select or clear the checkboxes for the MaxCompute projects to include in the collection.

      Important

      The default scope includes all MaxCompute projects in the current region that are attached to a workspace under the current tenant. After you modify the data scope, the metadata objects in Data Map are updated to match the new scope. This means that metadata for unselected projects will not be visible.

   3. Click Confirm to save the changes.

4. Configure metadata visibility:

   • In the Data Scope list, find the target project and click Permission Configurations in the Actions column.

   • Select a visibility policy based on your data governance requirements:

     • Public within Tenant: All members within the tenant can search for and view the metadata of this project.

     • Only members in the associated workspace can search and view.: Only members of specific workspaces can access the metadata of this project. This ensures data isolation.

DLF Default Crawler

The DLF Default Crawler collects metadata from Data Lake Formation (DLF) under your account.

1. On the collect metadata page, in the Built-in section, find the DLF Default Crawler card and click Details to view its basic information.

2. Switch to the Data Scope tab to view the list of DLF Catalogs that are within the collection scope and the number of tables they contain.

   By default, all accessible Catalogs are collected, including DLF and DLF-Legacy versions.

Create a custom crawler

1. On the collect metadata page, in the custom crawler section, click Create Metadata Collection.

2. Select Collection Type: On the type selection page, select the type of the target data source from which to collect metadata, such as Hologres or StarRocks.

3. Configure basic information and the resource group:

   • Basic configuration:

     • Select Workspace: Select the workspace where the data source is located.

     • Select Data Source: Select an existing target data source from the drop-down list. After you select a data source, the system automatically displays its details.

     • Name: Enter a name for the crawler. By default, the crawler name is the same as the data source name.

   • Resource group configuration:

     • Resource Group: Select a resource group to run the collection task.

     • Test Network Connectivity: This step is crucial. Click Test Network Connectivity to ensure that the resource group can successfully access the data source.

       Important

       • If the data source has whitelist-based access control enabled, you must configure the whitelist permissions. For more information, see Network connectivity solutions and General configuration: Add a whitelist.

       • If the data source does not use a whitelist, you must establish a network connection for it. For more information, see Resource group operations and network connectivity.

4. Metadata Collection Configurations:

   • Collection scope: Define the databases (Database/Schema) from which to collect metadata. If the data source has database-level granularity, the database associated with the data source is selected by default. You can also select other databases.

     Important

     • A database can only be configured in one crawler. If a database cannot be selected, it is already being collected by another crawler.

     • If you narrow the collection scope, metadata outside the new scope is no longer searchable in Data Map.

5. Intelligent Enhancement Settings and Collection Plan:

   • Intelligent Enhancement Settings (Beta):

     • AI-Enhanced Description: If you enable this feature, the system uses large models to automatically generate business descriptions for your tables and fields after collecting metadata. This greatly improves metadata readability and usability. After the collection is complete, you can go to the details page of a table object in Data Map to view the AI-generated information, such as table descriptions and field descriptions.

   • Collection Plan:

     • Trigger Mode: Select Manual or Cycle.

       • Manual: The crawler runs only when you manually trigger it. This is suitable for one-time or on-demand collection scenarios.

       • Cycle: Configure a scheduled task (such as monthly, daily, weekly, or hourly). The system will automatically update the metadata periodically.

         To configure a task with minute-level granularity, set the schedule to hourly and then select the desired minutes. For example, you can configure a task to run every 5 minutes.

         Important

         Only data sources in the production environment support periodic collection.

6. Save configuration: Click Save or Save and Run to create the crawler.

Manage custom crawlers

After a crawler is created, it appears in the custom crawler list. You can perform the following management operations:

• List operations: In the list, you can Run, Stop, or Delete a crawler. Use the Filter and Search functions at the top to quickly locate the target crawler.

  Important

  After a crawler is deleted, the metadata objects it collected are also removed from Data Map. These objects and their details will no longer be searchable or viewable. Proceed with caution.

• View details and logs: Click the name of the target crawler to go to its details page.

  • Basic Information: View all configuration items of the crawler.

  • Data Scope: View or Modify Data Scope.

    If viewed before a collection runs, the table count and last update time are empty.

    Modifying the scope is not supported for the following data sources: EMR Hive, CDH Hive, Lindorm, Elasticsearch, OTS, and AnalyticDB for Spark in AnalyticDB for MySQL.

  • Run Logs: Track the execution history of each collection task. You can view the task's start time, duration, status, and the volume of data collected. If a task fails, click View Logs to find information for troubleshooting and resolving the issue.

• Manually run a collection task: In the upper-right corner of the details page, click the Collect Metadata button to immediately trigger a collection task. This is useful if you want to immediately view a newly created table in Data Map.

Data tables

Task code

Data Map supports searching for and quickly locating task code. The following table describes the supported scope for code searches.

API assets

Data Map supports viewing the metadata of DataService Studio APIs, as detailed below:

AI assets

Data Map supports viewing and managing AI assets. It also provides an AI asset lineage feature to track the source, usage, and evolution of data and models. The following table describes the support for various AI assets.

Workspace

Data Map supports viewing workspace metadata, as detailed below: