DataWorks:View data lineages

E-MapReduce

• To collect metadata for a DataLake or custom cluster, configure EMR-HOOK in the cluster first. Without EMR-HOOK, Data Map cannot display lineages for that cluster. See Configure EMR-HOOK for Hive.

• Lineage is not available for Spark clusters created on the EMR on ACK page.

• Lineage is available for EMR Serverless Spark clusters.

• Lineage is not available for tasks developed using an EMR Presto node.

AnalyticDB for MySQL

AnalyticDB for MySQL supports real-time lineage when the metadata source is AnalyticDB for Spark. If the metadata source is AnalyticDB for Spark, data is collected automatically. To enable real-time lineage, set the following Spark parameter:

spark.sql.queryExecutionListeners = com.aliyun.dataworks.meta.lineage.LineageListener

SQL statements that do not generate lineage

Data Map cannot parse lineage from SQL that uses JOIN, UNION, the wildcard *, or subqueries. For example:

-- Lineage not supported: contains a wildcard (*)
INSERT INTO test SELECT * FROM test1, test2 WHERE test1.id = test2.id

-- Lineage not supported: contains a subquery
SELECT column1, column2 FROM table1 WHERE column3 IN (SELECT column4 FROM table2 WHERE column5 = 'value')

SQL statements that generate lineage

Lineage is captured for INSERT INTO, INSERT OVERWRITE, and CREATE TABLE AS SELECT when specific columns (not *) are listed:

-- Create a table from specific columns
CREATE TABLE test AS SELECT id, name FROM test1;

-- Insert specific columns with a filter
INSERT INTO test SELECT id, name FROM test1 WHERE name = 'test';

-- Overwrite using specific columns
INSERT OVERWRITE INTO db_name.test SELECT id, name FROM test1;

CDH

To display table lineage for CDH Spark SQL and CDH Spark nodes, add a Spark parameter to the relevant data transformation module in Management Center.

1. Log on to the DataWorks console. In the top navigation bar, select the region. In the left-side navigation pane, choose More > Management Center. Select your workspace from the drop-down list and click Go to Management Center.

2. In the left navigation pane, click Cluster Management and locate the target CDH cluster.

3. Click Edit Spark-related Parameter.

   

4. Add the following Spark parameter for the data transformation module. For example, to capture lineage for CDH Spark SQL and CDH Spark nodes under Operation Center > Auto Triggered Instances, add:

   Field	Value
   Spark Property Name	spark.sql.queryExecutionListeners
   Spark Property Value	com.aliyun.dataworks.meta.lineage.LineageListener

5. Click OK.

Lindorm

To display table lineage for Lindorm Spark and Lindorm Spark SQL nodes, add a Spark parameter in Management Center.

1. Log on to the DataWorks console. In the top navigation bar, select the region. In the left-side navigation pane, choose More > Management Center. Select your workspace and click Go to Management Center.

2. In the left navigation pane, click Computing Resource and locate your Lindorm computing resource.

3. Click Edit Spark-related Parameter.

4. Add the following Spark parameter for the data transformation module. For example, to capture lineage for Lindorm Spark and Lindorm Spark SQL nodes under Operation Center > Auto Triggered Instances, add:

   Field	Value
   Spark Property Name	spark.sql.queryExecutionListeners
   Spark Property Value	com.aliyun.dataworks.meta.lineage.LineageListener

5. Click OK.