All Products
Search
Document Center

Hologres:Data Map (Beta)

Last Updated:Feb 22, 2024

Data Map is a DataWorks module that you can use to manage the metadata and data assets of your business. You can use Data Map to view the details of metadata, view data lineage, and manage data categories. Data Map can help you search for, understand, and use Hologres data. This topic describes how to configure a crawler in Data Map to collect Hologres metadata as well as related operations.

Limits

  • Only Hologres V1.1 and later support Data Map. If the version of your instance is earlier than V1.1,

  • Data lineage information can be viewed 1 hour after the crawler is configured in Data Map.

  • The data lineage feature is available only in the following regions:

    China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), and Singapore.

Collect metadata

You can use the metadata collection feature to collect metadata from a Hologres data source to Data Map for centralized management. After the metadata is collected, you can search for and view the metadata in Data Map.

  1. Log on to the DataWorks console and go to the DataMap page. For more information, see Go to the DataMap page.

  2. In the top navigation bar, click Data Discovery.
  3. In the left-side navigation pane, choose Metadata Collection > Hologres.

  4. On the HologresMetadata Crawler page, click Create Crawler.

  5. In the Create Crawler dialog box, set the parameters in each step.

    1. Configure the basic information.

      1. In the Basic Information step, set the parameters as required. Basic Information step

        Parameter

        Description

        Crawler Name

        Required. The name of the crawler. You must set a unique name.

        Crawler Description

        The description of the crawler.

        Workspace

        The workspace of the data source from which you want to collect metadata.

        Data Source Type

        The type of the data source from which you want to collect metadata. The default value is Hologres and cannot be changed.

      2. Click Next.

    2. Select a Hologres data source.

      1. In the Select Collection Object step, select a data source from the Data Source drop-down list.

        You can collect metadata only from the Hologres instances that are associated with your workspace as compute engine instances. If no data source is available, click Create to create one. For more information, see Add a Hologres data source.

      2. Click Start Testing next to Test Crawler Connectivity. If the message The connectivity test is successful appears, the DataWorks metadata service can connect to the Hologres data source.

        Note

        If the message The connectivity test failed appears, you must find the cause of the connection error and troubleshoot the issue.

      3. Click Next.

    3. Configure an execution plan.

      In the Configure Execution Plan step, configure an execution plan.

      Valid values of the Execution Plan parameter are On-demand Execution, Monthly, Weekly, Daily, and Hourly. The execution plan that is generated varies based on the execution cycle. The system collects metadata from the Hologres data source based on the execution cycle that you specify. The following descriptions provide the details:

      • On-demand Execution: The system collects metadata from the Hologres data source based on your business requirements.

      • Monthly: The system automatically collects metadata from the Hologres data source once at a specific time on several specific days of each month.

        Important

        Specific months do not have the 29th, 30th, or 31st day. In these months, the system does not collect metadata from the Hologres data source on these dates. We recommend that you do not select the last few days of a month.

        The following figure shows that the system automatically collects metadata from the Hologres data source once at 09:00 on the 1st, 11th, and 21st day of each month. An expression is automatically generated for the Cron Expression parameter based on the values of the Date and Time parameters. Monthly

      • Weekly: The system automatically collects metadata from the Hologres data source once at a specific time on several specific days of each week.

        The following figure shows that the system automatically collects metadata from the Hologres data source once at 03:00 on Sunday and Monday of each week. An expression is automatically generated for the Cron Expression parameter based on the values of the Week and Time parameters. WeeklyIf the Time parameter is not set, the system automatically collects metadata from the Hologres data source once at 00:00:00 on the specific days of each week.

      • Daily: The system automatically collects metadata from the Hologres data source once at a specific time of each day.

        The following figure shows that the system automatically collects metadata from the Hologres data source once at 01:00 each day. An expression is automatically generated for the Cron Expression parameter based on the values of the Time parameter. Daily

      • Hourly: The system automatically collects metadata from the Hologres data source once on the N × 5th minute of each hour.

        Note

        For a Hologres metadata collection task that is run each hour, you can set the Minute value to a multiple of 5 minutes.

        The following figure shows that the system automatically collects metadata from the Hologres data source from the 5th and 10th minutes of each hour. An expression is automatically generated for the Cron Expression parameter based on the values of the Minutes parameter. Hourly

      • Click Next.

    4. Confirm the settings of the crawler.

      In the Confirm Information step, check the information that you specified.

  6. Click Confirm.

  7. On the HologresMetadata Crawler page, you can view the information about your crawler and manage your crawler.

    The following descriptions show the information that you can view and the operations that you can perform:

    • You can view the status and execution plan of the crawler. You can also view the time when the last execution started, the amount of time consumed for the last execution, the average amount of time consumed, the number of updated tables in the last execution, and the number of created tables in the last execution.

    • You can click Details, Edit, Delete, Run, or Stop in the Actions column to perform the desired operation.

      • Details: View the crawler name and the data source and execution plan configured for the crawler.

      • Edit: Modify the configurations of the crawler.

      • Delete: Delete the crawler.

      • Run: Run a task to collect metadata from the Hologres data source. The Run button is available only if the Execution Plan parameter is set to On-demand Execution.

      • Stop: Stop the crawler. The Stop button is displayed only if a crawler is in the Pending state.

Other supported operations

  • View overall data

    On the Overview tab of the DataMap page, you can view the statistical and table information of all Hologres databases that have the data crawler configured in a region. For more information, see View resource information. Overall data

  • Search for tables

    Data Map allows you to search for tables by table name, table description, field name, and field description. You can also filter tables by category, project, and database. For more information, see Search for tables. Table search

  • View the details of a table

    You can go to the details page of a table and view the details of the table, such as the basic information, output information, and lineage information of the table. For more information, see View the details of a table.

    • You can view the metadata of a table. Metadata information

    • You can view table lineage and field lineage information. Lineage information