DataWorks Data Map provides topological visualization and full-lifecycle management for AI assets through a unified metadata hub, improving the efficiency of asset governance. This topic describes how to view AI assets in Data Map.
Background
Data Map integrates metadata from PAI AI assets to support global search and management of your core AI assets. A unified interface lets you efficiently discover and manage these resources and use the lineage service to track the origin, usage, and evolution of data and models.
The AI asset lineage service provides an intuitive way to track the data and model lifecycle, which helps you manage and optimize your AI assets. The details page for each AI asset object provides an entry point to view and analyze its lineage information.
For example, a typical AI asset lineage graph shows the complete chain from input files, such as Object Storage Service (OSS) storage paths, and datasets, such as MaxCompute tables, through a PAI model training task in Model Gallery, which generates output files, such as model adapter files and labeled dataset files, an AI model, and an online model service.
The following are the main use cases for the AI asset lineage service:
AI asset governance: Detailed asset lineage information helps you understand the origin and usage of your AI assets. This not only helps improve the quality of data and models but also ensures that your AI practices meet compliance requirements and supports more precise data management and decision-making.
Model traceability: In the context of Responsible AI practices, maintaining the transparency of AI models is critical. The AI asset lineage service allows you to track the datasets, feature engineering methods, and parameter tuning processes used in model training. This traceability is essential for meeting regulatory requirements, validating experimental results, and conducting model audits.
Troubleshooting and optimization: When an AI service experiences performance issues, asset lineage information can help you quickly locate the root cause. For example, if a model's prediction accuracy suddenly drops, the cause may be a change in upstream data processing. With the lineage graph, you can rapidly identify and resolve these issues.
Improving resource utilization: Understanding the dependencies between tasks allows you to allocate computing resources more effectively, avoid redundant computations, and reduce costs. In large-scale experiments, understanding task and data dependencies through lineage information helps you identify which tasks can run in parallel, enabling more efficient resource utilization and processing.
Enhancing collaboration efficiency: In large organizations, multiple teams may share the same infrastructure for research. Clear task lineage information promotes cross-team collaboration and accelerates the innovation process.
Prerequisites
You must create a workspace for the new version of Data Studio. Select one of the following editions based on your requirements:
If you have created a PAI workspace, you can search for and view the corresponding types of AI assets. When you create the workspace, you must select Create AI Workspace with Same Name.
DataWorks Standard Edition: To use standard AI asset lineage services, you must activate DataWorks Standard Edition.
DataWorks Professional Edition: If you need lineage reporting capabilities for Deep Learning Containers (DLC) model training tasks and workflow tasks, you must activate DataWorks Professional Edition.
Supported AI assets
DataWorks allows you to search for and view the metadata details of the following AI assets in Data Map:
Collect AI assets
After you enable PAI for your workspace, Data Map automatically collects the asset information from your AI workspace.
Search for AI assets
You can use the search feature to view the basic information and lineage of all AI assets in the current workspace.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left-side navigation pane to go to the Search page.In the Type section on the left side of the page, select the AI asset type that you want to view, and enter keywords in the search box to find the corresponding AI assets.
View AI asset lineage
The search page in DataWorks Data Map allows you to view the lineage of AI assets. You can also navigate to PAI to view AI asset details.
The dataset lineage feature in Data Map provides aggregated version lineage information. You can view the lineage of all versions of an AI asset. The aggregated version lineage consolidates information from multiple versions, and you can view the lineage of a specific version or all versions.
Dataset lineage
Lineage generation and reporting
When you generate a dataset or perform operations on a dataset through the following methods, the system automatically triggers lineage generation and reporting.
Method 1: Register a storage path as a dataset
When you register a storage path as a dataset, the following lineage information is generated. The system automatically reports the following lineage information.
Data Source | Description |
When the data source of the registered dataset is OSS, a lineage record from OSS to the dataset is generated. | |
|
Method 2: Register a data table as a dataset
When the data source of the registered dataset is a MaxCompute table, a lineage record from the MaxCompute table to the dataset is generated. The lineage information is structured as follows:
If you generate a MaxCompute table through multiple SQL tasks and register the table as a PAI dataset, the lineage information of this dataset allows you to trace back to all related SQL tasks that generated the table.
Method 3: Label a dataset
When you label a dataset in iTAG, and the Create a labeling job is created, the system automatically reports the following lineage information.
View dataset lineage
You can view dataset lineage through the following entry points.
Entry 1: View dataset lineage in Data Map
You can view dataset lineage information in Data Map. Perform the following steps:
Go to the dataset list page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left-side navigation pane to go to the Search page.In the Type section on the left side of the page, select Dataset. Use the Data Source Type, Workspace, and other options in the Filter Conditions section to obtain the dataset list that you want to view.
View the dataset lineage.
In the dataset list, find the target dataset, and then click View Lineage on the right side to view its lineage information.
On the lineage details page, the left side displays the upstream associated OSS file node, and the right side highlights the center node of the current dataset and its version information in orange.
On the dataset lineage details page, you can perform the following operations:
Lineage Time Range: In area ①, select a lineage time range to view the lineage of the dataset over the past few days.
Dataset Vers.: In area ②, switch the dataset version to view the lineage of the corresponding version.
Other operations: Right-click the dataset, and select View Details (navigate to PAI to view the dataset details) or View Lineage from the context menu.
Entry 2: View dataset lineage in PAI
In PAI, you can view the lineage information of a dataset in the details section of a specified version. Perform the following steps:
Log on to the PAI console. Switch to the target region. In the left-side navigation pane, choose Workspaces, and click the name of the target workspace to go to the workspace.
In the left-side navigation pane, choose .
Find the dataset that you created in the Basic Dataset section on the Custom Dataset tab.
Click the dataset name, and in the details section of a specified version, click View Lineage to go to the PAI Bloodline analysis page, where you can view the lineage of the current dataset version.
NoteOn the PAI lineage analysis page, right-click any node in the lineage graph, and select View Lineage to navigate to Data Map and view the upstream and downstream lineage of the node.
AI model lineage
Lineage generation and reporting
When you generate an AI model or perform operations on an AI model through the following methods, the system automatically triggers lineage generation and reporting.
Method 1: Run a model training task
After a model training task submitted in Model Gallery is completed, a lineage record between the model and the training task is generated. The system automatically reports the following lineage information.
Method 2: Register Model
After you register a model on the model management page, a lineage record from the model storage path to the model is generated. The system automatically reports the following lineage information.
Method 3: Deploy a model service
After you deploy a registered model to EAS, a lineage record from the model to the model service is generated. The system automatically reports the following lineage information.
View AI model lineage
You can view AI model lineage through the following entry points.
Entry 1: View AI model lineage in Data Map
You can view AI model lineage information in Data Map. Perform the following steps:
Go to the AI model list page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left-side navigation pane to go to the Search page.In the Type section on the left side of the page, select AI Model. Use the Workspace option in the Filter Conditions section to obtain the AI model list that you want to view.
View the AI model lineage.
In the AI model list, find the target AI model, and then click View Lineage on the right side to view its lineage information.
On the AI model lineage details page, you can perform the following operations:
Lineage Time Range: In area ①, select a lineage time range to view the lineage of the AI model over the past few days.
AI Model Vers.: In area ②, switch the AI model version to view the lineage of the corresponding version.
Other operations: Right-click the AI model, and select View Details (navigate to PAI to view the AI model details) or View Lineage from the context menu.
Entry 2: View AI model lineage in PAI
In PAI, you can view the lineage information of an AI model in the details section of a specified version. Perform the following steps:
Log on to the PAI console. Switch to the target region. In the left-side navigation pane, choose Workspaces, and click the name of the target workspace to go to the workspace.
In the left-side navigation pane, choose .
Find the model that you registered on the Model page.
Click the latest version number to go to the Model Version information page.
In the Version Overview section, click View Lineage to go to the PAI Bloodline analysis page, where you can view the lineage of the current AI model version.
NoteOn the PAI lineage analysis page, right-click any node in the lineage graph, and select View Lineage to navigate to Data Map and view the upstream and downstream lineage of the node.
Algorithm task lineage
Lineage generation and reporting
When you run related tasks through the following methods, the system automatically triggers lineage generation and reporting.
For tasks submitted through DLC distributed training, you must manually report lineage information by integrating an SDK and configure the inputs and outputs based on the actual task. This approach is suitable for users with strong technical capabilities and mature business practices. Otherwise, the accuracy of lineage information may be affected. To use this feature, contact your account manager to be added to the allowlist. For more information, see Create a model training job.
Method 1: Run a Training Task (algorithm component task)
After a model training task submitted in Model Gallery is completed, a lineage record between the model and the training task is generated. The system automatically reports the following lineage information.
Method 2: Run a Workflow Task
After a workflow task created in PAI is completed, lineage relationships are generated among the input tables or storage paths, output models or output datasets, and the PAIFlow task.
The lineage generated by PAI workflow tasks run through the following methods is automatically collected and reported:
Tasks submitted by using PAIFlow OpenAPI. For more information, see List of operations by function.
Tasks manually run in Designer. For more information, see Build and debug a model.
Tasks periodically triggered through DataWorks periodic scheduling. For more information, see Schedule a PAI Designer pipeline with DataWorks.
Method 3: Run a distributed training task
When you create and run a distributed training task, lineage relationships are generated among the input table storage paths, output models or output datasets, and the DLC task. For tasks submitted through DLC distributed training, you can manually report lineage information.
This approach is suitable for users with strong technical capabilities and mature business practices. Otherwise, the accuracy of lineage information may be affected.
View algorithm task lineage
You can view algorithm task lineage through the following entry points.
Entry 1: View algorithm task lineage in Data Map
The algorithm task lineage feature in Data Map allows you to view the lineage of a task. You can perform the following operations to view the lineage:
Go to the algorithm task list page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left-side navigation pane to go to the Search page.In the Type section on the left side of the page, select Algorithm Task. Use the Workspace option in the Filter Conditions section to obtain the algorithm task list that you want to view.
View the algorithm task lineage.
In the algorithm task list, find the target algorithm task, and then click View Lineage on the right side to view its lineage information.
On the algorithm task lineage details page, you can perform the following operations:
Lineage Time Range: Select a lineage time range to view the lineage of the algorithm task over the past few days.
Other operations: Right-click the algorithm task, and select View Details (navigate to PAI to view the algorithm task details) or View Lineage from the context menu.
Entry 2: View algorithm task lineage in PAI
In PAI, you can view the lineage information of an algorithm task in the basic information section of the task details page. Perform the following steps:
Log on to the PAI console. Switch to the target region. In the left-side navigation pane, choose Workspaces, and click the name of the target workspace to go to the workspace.
In the left-side navigation pane, choose .
On the Tasks page, find the tab where your algorithm task is located (Distributed Training (DLC), Algorithm Component Task, or Workflow Task tab).
Click the task name to go to the task details page.
In the Basic Information section, click View Lineage to go to the PAI Bloodline analysis page, where you can view the lineage of the algorithm task.
NoteOn the PAI lineage analysis page, right-click any node in the lineage graph, and select View Lineage to navigate to Data Map and view the upstream and downstream lineage of the node.
Model service lineage
Lineage generation and reporting
After you deploy a registered model to EAS, the system automatically establishes a lineage relationship between the model and the model service and reports the lineage information.
View model service lineage
You can view model service lineage through the following entry points.
Entry 1: View model service lineage in Data Map
The model service lineage feature in Data Map allows you to view the lineage of a model service. You can perform the following operations to view the lineage:
Go to the model service page.
Log on to the DataWorks console. In the target region, click in the left-side navigation pane. On the page that appears, click Go to Data Map.
Click the
icon in the left-side navigation pane to go to the Search page.In the Type section on the left side of the page, select Model Service. Use the Status and Resource Group Name options in the Filter Conditions section to obtain the model service list that you want to view.
View the model service lineage.
In the model service list, find the target model service, and then click View Lineage on the right side to view its lineage information.
On the model service lineage details page, you can perform the following operations:
Lineage Time Range: Select a lineage time range to view the lineage of the model service over the past few days.
Other operations: Right-click the model service, and select View Details (navigate to PAI to view the model service details) or View Lineage from the context menu.
Entry 2: View model service lineage in PAI
In PAI, you can view the lineage information of a model service in the basic information section of the model service details page.
Log on to the PAI console. Switch to the target region. In the left-side navigation pane, choose Workspaces, and click the name of the target workspace to go to the workspace.
In the left-side navigation pane, choose .
Find the Inference Service tab on the Elastic Algorithm Service (EAS) page for the model that you created.
Click the task name to go to the task overview page.
In the Basic Information section, click View Lineage to go to the PAI Bloodline analysis page, where you can view the lineage of the model service.
NoteOn the PAI lineage analysis page, right-click any node in the lineage graph, and select View Lineage to navigate to Data Map and view the upstream and downstream lineage of the node.