You can collect the table schema and lineage to DataMap, which displays the inner structure and association relationships of a table. This topic describes how to create a crawler and collect Tablestore metadata to DataWorks. You can view the collected metadata on the DataMap page.

  1. Go to the Data Discovery page.
    1. Log on to the DataWorks console.
    2. In the left-side navigation pane, click Workspaces.
    3. In the top navigation bar, select a region as required, find the workspace where you want to create a crawler, and then click Data Analytics in the Actions column.
    4. On the DataStudio page, click the DataWorks icon icon in the upper-left corner and choose All Products > Data governance > DataMap. The homepage of Data Map appears.
    5. In the top navigation bar, click Data Discovery.
  2. In the left-side navigation pane, click OTS.
  3. On the OTSMetadata Crawler page, click Create Crawler.
  4. In the Create Crawler dialog box, perform the following steps:
    1. In the Basic Information step, configure the parameters.
      Basic information
      Parameter Description
      Crawler Name Required. This parameter specifies the name of the crawler. You must specify a unique name.
      Crawler Description The description of the crawler.
      Workspace The workspace of the data source from which you want to collect metadata.
      Connection Type The type of data source from which you want to collect metadata. The default value is OTS and cannot be changed.
    2. Click Next.
    3. In the Select Collection Object step, select a data source from the Connection drop-down list.
      If the drop-down list does not contain the data source that you need, click Create Connection to go to the data source page to create a data source. For more information, see Configure a Tablestore connection.
    4. Click Test Crawler Connectivity.
      If the database is configured with a whitelist, you must add the IP address of the workspace based on the region where the workspace resides to the whitelist.
    5. If the message The connectivity test has been passed appears, click Next.
      If the message The connectivity test failed appears, check whether you have configured a valid data source.
    6. In the Configure Execution Plan step, specify Execution Plan.
      Valid values of the Execution Plan parameter: On-demand Execution, Monthly, Weekly, Daily, and Hourly. Different running plans are generated based on different running cycles. The system collects Tablestore metadata of the data source based on the running cycle that you specify. The following descriptions explain each value and provide examples:
      • On-demand Execution: The system collects Tablestore metadata based on your business requirements.
      • Monthly: The system automatically collects Tablestore metadata each day on several specific days of each month.
        Notice Some months do not have the 29th, 30th, or 31st day, so the system does not collect Tablestore metadata on these dates. We recommend that you do not select dates at the end of months.
        The following figure shows that the system collects Tablestore metadata once a day at 09:00 on the 1st, 11th, and 21th day of each month. CRON Expression is automatically generated based on the values of Date and Time.Monthly
      • Weekly: The system automatically collects Tablestore metadata once a day at a specific time point several days a week.
        The following figure shows that the system automatically collects Tablestore metadata at 03:00 on Sunday and Monday of each week.WeeklyIf the Time parameter is not specified, the system automatically collects Tablestore metadata at 00:00:00 on the specific days of each week.
      • Daily: The system automatically collects Tablestore metadata at a specific time point of each day.
        The following figure shows that the system automatically collects Tablestore metadata at 01:00 each day.Daily
      • Hourly: The system automatically collects Tablestore metadata at N × 5 minutes each hour.
        Note For a Tablestore metadata collection task that runs each hour, you can select a time point that is a multiple of five minutes.
        The following figure shows that the system automatically collects Tablestore metadata at the 5th and 10th minute of each hour.Hourly
    7. Click Next.
    8. In the Confirm Information step, check the information that you specified and click OK.
  5. On the OTSMetadata Crawler page, you can view the information about your crawler and manage your crawler.View the crawler
    The following description shows the information that you can view and the operations that you can perform:
    • You can view Status, Execution Plan, Last Run At, Last Consumed Time, Average Running Time, Updated Tables in Last Run, and Added Tables in Last Run of your crawler.
    • You can click Details, Edit, Delete, Run, or Stop in the Actions column to perform the required operations.
      • Details: View Crawler Name, Connection Type, and Execution Plan of the crawler.
      • Edit: Modify the configurations of the crawler.
      • Delete: Delete the crawler.
      • Run: Run the task to collect Tablestore metadata. The Run entry is available only when Execution Plan is set to On-demand Execution.
      • Stop: Stop running the crawler.

Result

After Tablestore metadata is collected, switch back to the previous page and click All Data in the top navigation bar. On the page that appears, click the OTS tab in the upper part. On the OTS tab, you can view the table that stores the collected Tablestore metadata.View the collected Tablestore metadata

Click table name, workspace, or database to view the related details.

Example 1: View the details of the mysql_ots table.View the details of a table
Example 2: View all tables in the datax-bvt database.View all tables in a database