The data lineage information includes the lineages between tables and the lineages between fields. You can use the data lineage feature to trace the source of data and manage data based on your business requirements. You can also use the data lineage feature to analyze the impacts of a job failure on upstream and downstream jobs. Hologres is deeply integrated with DataWorks. You can use the Data Map service of DataWorks to manage data lineages in Hologres.
Background information
Data Map is a DataWorks service used to manage data directories of enterprises based on metadata. The Data Map service allows you to perform various operations, such as viewing the details of metadata and managing data lineages and data categories. Data Map can help you search for, understand, and use data. For more information, see Overview.
Limits
Only Hologres V1.1 and later support the Data Map service. If the version of your instance is earlier than V1.1, manually upgrade your Hologres instance in the Hologres console or join a DingTalk group to apply for an instance upgrade. For more information about how to manually upgrade a Hologres instance, see Instance upgrades. For more information about how to join a DingTalk group, see Obtain online support for Hologres.
The data lineage feature is supported only in DataWorks Standard Edition or a more advanced edition.
Data lineage information can be viewed 1 hour after the Hologres metadata crawler is configured in Data Map.
The data lineage feature is available only in the following regions:
China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), Singapore, and China East 2 Finance.
View data lineage information of Hologres
This section describes how to view the data lineages between Hologres tables in DataWorks Data Map.
Collect and view Hologres metadata.
Use the metadata collection feature to import metadata from a Hologres data source to Data Map for centralized management. For more information, see Metadata collection.
After the metadata is collected, you can go to the data overview page and view the statistical information of all Hologres databases and tables for which the data crawler feature is configured in a region. For more information, see View overall data.
If you want to search for a table, follow the instructions in Query and manage common data.
View data lineage details.
Find the desired table and click the name of the table to go to the details page of the table. The table details page displays table information such as the basic information, output information, and data lineage information. For more information, see View the details of a table.
The following figure shows the details page of a Hologres internal table. You can view the metadata information on the details page and the data lineage information on the Lineage tab.
View lineage information between Hologres and MaxCompute
DataWorks Data Map allows you to view the data lineage information between MaxCompute and Hologres. You can click the link next to External Table to view the information about the MaxCompute table to which the Hologres foreign table is mapped, as shown in the following figure.
You can click the link to go to the details page of the mapped MaxCompute table only after the project to which the MaxCompute table belongs is associated with the desired DataWorks workspace.
You can view the table lineage information on the Table Lineage subtab. As shown in the following figure, data is written to the MaxCompute table by using the Hologres foreign table based on DataWorks. You can also view the field lineage information on the Field Lineage subtab.
View lineage information between Hologres and Flink
You can view the lineage information between Hologres and Flink in the Realtime Compute for Apache Flink console. For more information, see View data lineage.