DataWorks:A practical guide to the data lineage OpenAPI

Get a table entity ID from the console

For a small number of known tables, copy the ID directly from the interface.

1. Go to the Data Map module in DataWorks.

2. Search for and open the details page of the target table.

3. In the Table Basic Information panel on the left, find the Entity ID and copy it.

   

Get a field entity ID from the console

1. On the table's details page, switch to the Lineage tab and select Field Lineage.

2. In the field lineage graph, click the field node.

3. In the field's details panel on the right, find the Entity ID and copy it.

   

Get entity IDs in batches using the API

For large-scale analysis, call these APIs instead of using the console:

• Tables: ListTables — returns a list of tables in Data Map along with their entity IDs

• Fields: ListColumns — returns fields for a specific table along with their entity IDs

Parameters

Test the API interactively in the OpenAPI Explorer.

Example 1: Query downstream lineage

Find all tables that consume test_table:

SrcEntityId:            maxcompute-table:::test_project::test_table
NeedAttachRelationship: true

To filter results to downstream tables whose names contain "report", add:

DstEntityName: report

Example 2: Query upstream lineage

Find all tables and tasks that produced test_table:

DstEntityId:            maxcompute-table:::test_project::test_table
NeedAttachRelationship: true

To filter upstream results by name, add SrcEntityName.

Response structure

A successful response contains a list of lineage relationship objects. Each object has this structure:

{
  "SrcEntity": {
    "Id": "maxcompute-table:::test_project::table_from",
    "Name": "table_from",
    "Attributes": {
      "rawEntityId": "maxcompute-table:::test_project::table_from"
    }
  },
  "DstEntity": {
    "Id": "maxcompute-table:::test_project::table_to",
    "Name": "table_to",
    "Attributes": {
      "project": "test_project",
      "region": "cn-shanghai",
      "table": "table_to"
    }
  },
  "Relationships": [
    {
      "Id": "123456789:maxcompute-table.test_project.table_from:maxcompute-table.test_project.table_to:maxcompute.SQL.76543xxx",
      "CreateTime": 1761089163548,
      "Task": {
        "Id": "76543xxx",
        "Type": "dataworks-sql",
        "Attributes": {
          "engine": "maxcompute",
          "channel": "1st",
          "taskInstanceId": "12345xxx",
          "projectId": "123456",
          "taskId": "76543xxx"
        }
      }
    }
  ]
}

Key fields:

• SrcEntity / DstEntity: The upstream and downstream data entities. Use the Id field to call GetTable or GetColumn for full metadata.

• Relationships: The edges connecting the two entities. Each edge describes the task that wrote data from source to destination.

• Task: The computing or scheduling task that created this lineage edge. For DataWorks scheduling tasks, Task.Attributes includes taskId and taskInstanceId. Pass these to GetTask to retrieve the task definition and run status.

ListLineages returns one hop at a time. To trace multi-hop lineage — for example, finding the root source of a table that is itself derived from other tables — call the API iteratively, using the SrcEntity.Id or DstEntity.Id from each response as the input for the next call.

Set up the project

1. Make sure JDK 8 or later is installed.

2. Add the following dependency to your pom.xml. Replace ${latest.version} with the version from the SDK page.

<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>dataworks_public20240518</artifactId>
    <version>${latest.version}</version>
</dependency>

Query lineage

The following example initializes the client, queries both upstream and downstream lineage for a specified table, and prints a human-readable summary.

All requests use the same pattern: set either SrcEntityId (for downstream) or DstEntityId (for upstream), enable NeedAttachRelationship, and iterate over the response to extract entity names and task IDs.

import java.util.List;
import java.util.Map;

import com.aliyun.dataworks_public20240518.Client;
import com.aliyun.dataworks_public20240518.models.GetTableRequest;
import com.aliyun.dataworks_public20240518.models.GetTableResponse;
import com.aliyun.dataworks_public20240518.models.LineageEntity;
import com.aliyun.dataworks_public20240518.models.LineageRelationship;
import com.aliyun.dataworks_public20240518.models.LineageTask;
import com.aliyun.dataworks_public20240518.models.ListLineagesRequest;
import com.aliyun.dataworks_public20240518.models.ListLineagesResponse;
import com.aliyun.dataworks_public20240518.models.ListLineagesResponseBody.ListLineagesResponseBodyPagingInfo;
import com.aliyun.dataworks_public20240518.models.ListLineagesResponseBody.ListLineagesResponseBodyPagingInfoLineages;
import com.aliyun.dataworks_public20240518.models.Table;
import com.aliyun.tea.TeaException;

public class LineageQuerySample {
    // Replace with your actual entity ID from Data Map
    private static final String TARGET_ENTITY_ID = "maxcompute-table:::test_project::test_table";

    public static void main(String[] args) throws Exception {
        // Initialize the client using credentials from environment variables
        com.aliyun.teaopenapi.models.Config config = new com.aliyun.teaopenapi.models.Config()
            .setAccessKeyId(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID"))
            .setAccessKeySecret(System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET"))
            .setEndpoint("dataworks.cn-shanghai.aliyuncs.com");  // Replace with your region endpoint
        Client client = new Client(config);

        System.out.println("=== Downstream lineage (tables that consume " + TARGET_ENTITY_ID + ") ===");
        queryLineage(client, TARGET_ENTITY_ID, null);

        System.out.println("\n=== Upstream lineage (tables that produce " + TARGET_ENTITY_ID + ") ===");
        queryLineage(client, null, TARGET_ENTITY_ID);
    }

    /**
     * Queries one hop of lineage for a given entity.
     * @param srcEntityId  Set to query downstream lineage; leave null to query upstream.
     * @param dstEntityId  Set to query upstream lineage; leave null to query downstream.
     */
    static void queryLineage(Client client, String srcEntityId, String dstEntityId) throws Exception {
        ListLineagesRequest request = new ListLineagesRequest()
            .setSrcEntityId(srcEntityId)
            .setDstEntityId(dstEntityId)
            .setNeedAttachRelationship(true);

        try {
            ListLineagesResponse response = client.listLineages(request);
            ListLineagesResponseBodyPagingInfo pagingInfo = response.getBody().getPagingInfo();

            if (pagingInfo == null || pagingInfo.getLineages() == null || pagingInfo.getLineages().isEmpty()) {
                System.out.println("  No lineage relationships found.");
                return;
            }

            for (ListLineagesResponseBodyPagingInfoLineages lineage : pagingInfo.getLineages()) {
                LineageEntity src = lineage.getSrcEntity();
                LineageEntity dst = lineage.getDstEntity();
                System.out.printf("  %s --> %s%n",
                    src != null ? src.getName() : "unknown",
                    dst != null ? dst.getName() : "unknown");

                // Print the task that created each lineage edge
                if (lineage.getRelationships() != null) {
                    for (LineageRelationship rel : lineage.getRelationships()) {
                        LineageTask task = rel.getTask();
                        if (task != null && task.getAttributes() != null) {
                            Map<String, String> attrs = task.getAttributes();
                            System.out.printf("    created by task %s (instance: %s)%n",
                                attrs.getOrDefault("taskId", "N/A"),
                                attrs.getOrDefault("taskInstanceId", "N/A"));
                        }
                    }
                }
            }
        } catch (TeaException e) {
            System.err.printf("API error: %s - %s%n", e.getCode(), e.getMessage());
            throw e;
        }
    }
}

Expected output:

=== Downstream lineage (tables that consume maxcompute-table:::test_project::test_table) ===
  test_table --> sales_report
    created by task 76543xxx (instance: 12345xxx)

=== Upstream lineage (tables that produce maxcompute-table:::test_project::test_table) ===
  orders_raw --> test_table
    created by task 65432yyy (instance: 11111yyy)

Once you have the entity IDs from the response, call GetTable for full table metadata or GetTask for the task definition.