Data lineage shows the relationships between tables or between fields. It helps you trace and manage data for your business and analyze the impact on upstream and downstream jobs when a job fails. Hologres is deeply integrated with DataWorks. You can use the Data Map module in DataWorks to manage Hologres data lineage.
Background information
DataWorks Data Map is an enterprise data catalog management module built on metadata. It provides features such as viewing metadata details, data lineage, and data category management. Data Map helps you find, understand, and use data more effectively. For more information, see Data Map overview.
Limits
-
Only Hologres V1.1 or later supports the Data Map feature. If your instance runs a version earlier than V1.1, you must upgrade it. For assistance, see Common upgrade preparation failure errors or join the Hologres DingTalk group to provide feedback. For more information, see How do I get more online support?.
-
Only DataWorks Standard Edition or higher lets you view lineage information.
-
Lineage information appears in Data Map one hour after you configure the Hologres metadata collector.
View Hologres data lineage
You can use DataWorks Data Map to view lineage relationships between Hologres tables. Perform the following steps:
-
Ingest and connect Hologres metadata.
You can use the metadata acquisition feature to import metadata from your Hologres data source into Data Map for centralized management. For more information, see Metadata acquisition.
After metadata acquisition is complete, you can view statistics and table information for Hologres databases for which collectors are configured in the current region on the Data Overview page. For more information, see Data Overview.
To find a specific table, see Metadata retrieval.
-
View lineage details.
After you find the target table, click its name to go to the table details page. On this page, you can view basic information, output information, and lineage information. For more information, see View table details.
The following image shows an example of a Hologres internal table. On this page, you can view its metadata and switch to the Lineage tab to view its lineage.
View lineage with MaxCompute
DataWorks Data Map also supports viewing the data lineage between MaxCompute and Hologres. In the following example, you can click the Hologres foreign table link to view the details of the corresponding MaxCompute table.
To navigate from a foreign table link to the corresponding MaxCompute table details page, the MaxCompute project that contains the table must be attached to your target DataWorks workspace.
On this page, you can view table lineage information. For example, the following image shows that data is written from a Hologres foreign table to a MaxCompute table using the scheduling module in DataWorks. You can also switch to the Field Lineage tab to view field-level lineage.
View lineage with Flink
You can view the data lineage between Hologres and Flink in the Flink console. For more information, see View lineage relationships.