Dataphin:MaxCompute physical tables and fields

Area

Description

① Summary

This section displays summary information about the table, such as its type, environment, name, tags, and description. You can also perform these operations:

• Search for other assets: Quickly search for and switch to the details page of other assets.

• View asset details: If the current object is listed in the asset catalog, you can go to the catalog details page to view the listing.

• View production/development object: Quickly switch between the object details in the corresponding production and development environments.

• Tags: Displays the tag values configured for the current asset. To modify tags, click Edit.

  • Each tag value can be up to 128 characters long.

  • Each asset can have up to 20 tag values.

  • A super administrator can modify tags for all asset types. An owner can modify tags for assets they own. A project administrator can modify tags for physical tables in projects they manage.

• Subscribe to Changes: You can use this operation only after you purchase the Asset Operations, Metadata Collection, and Metadata Management features. Select the data table to which you want to subscribe. In the New Subscription dialog box, configure the subscription content, notification method, and frequency. You can subscribe only to tables for which you have view permissions. Otherwise, the subscription fails. For more information, see Add a subscription.

• Unsubscribe: Unsubscribe from a data table to stop receiving notifications about its status changes.

• Add to Favorites: Click to add an asset to or remove it from your favorites. The 30 most recently favorited assets are available in Asset List > My History for quick access. You can also view all your favorited assets in the Personal data center. For more information, see View and manage my favorite assets.

• Analyze: Opens the Notebook page and automatically creates a Notebook task. For more information, see Create a Notebook.

• Request Permissions: Click to go to the permission request page for the current table. For more information, see Request, renew, and release table permissions.

• Report Data Quality Issue: Report data quality issues, such as unstable output or inaccurate content, to the quality owner. This action notifies the relevant owner to resolve the issue, helping improve asset availability and health. For configuration details, see Add and manage issue lists.

  This feature requires the data quality module. You can then track the processing flow and results in the issue list of the data quality module.

• Generate SELECT statement: Click to generate a query statement for the current table. You can choose whether to add escape characters. You can copy the query statement and run it on the ad hoc query or analysis page.

  

• View DDL Statement: Click to view the DDL statement for the current data table in the tenant's compute engine. You can also select a Data Source Type and click Generate DDL Statement to generate a DDL statement for creating a table with the same schema in the specified data source. If you select Automatically add escape characters, the generated DDL statement includes the appropriate escape characters for the selected source type to reduce issues such as incorrect escaping of reserved keywords.

  

• Edit Table: Click to go to the Develop > Table Management page to edit the DDL information of the table. For more information, see Create an offline physical table.

• Export Fields: Export the table's field information as a CSV file for analysis and use by developers or business personnel.

• View Transfer Records: Displays the 100 most recent owner transfer records for the current data table.

• View Permission List: View the permission information for the data table.

• Refresh Metadata: If a data table was not created on the Dataphin platform, or if you cannot find a newly created table due to metadata retrieval delays, you can click Refresh Metadata to fetch the latest metadata. This action refreshes the metadata of the specified data table in the Dataphin system. 

② Details

This section displays table, field, and partition information.

• Table Details: Displays the properties of the table, including data section, subject domain, project, highest sensitivity level (requires the Data Security feature), whether the table is partitioned, whether it is a manually created table on the analysis platform, whether it is a lakehouse table, lakehouse format, table storage mode, storage type, storage format, storage size, lifecycle, and location.

  • Project: The project to which the table belongs. Click the project name to go to the asset details page for that project.

  • Highest sensitivity level: The highest sensitivity level among the table's fields. This level indicates the data's confidentiality. Data classification levels range from low to high: L1, L2, L3, and L4 (Public to Top Secret), and custom data classification levels are also supported.

  • Table storage mode: This information is collected from the table's source (compute source) if the data table is a lakehouse table and its format is Hudi.

  • Storage type: The storage type of the table, which can be an internal table or an external table.

  • Storage size: The actual storage size of the table. This is updated on a T+1 basis.

  • Lifecycle: The retention period of the table, in days.

  • Location: For external tables from MaxCompute and Hive compute sources, you can view the table's storage path, for example, hdfs://node01.exp5:8020/user/hive/warehouse.

• Field Information: Includes the field name, description, data type, associated standard (requires the Data Standard module), sample data (displayed only when the data sampling feature is enabled), data category (requires the Data Security module), data classification (requires the Data Security module), and popularity information. You can also search for fields, filter fields, and view the lineage of fields.

  • View lineage: Click the lineage icon in the Actions column to view the field-level lineage for the specified field.

  • Search and filter: You can search for fields by name or description. You can also filter fields by data category or data classification (requires the Data Security feature).

• Partition Information: Displays the partition information of the data table. Multi-level partitions are displayed as a single string, with each level separated by a forward slash (/). For example: ds=20221001/pt1=a/pt2=b.

  Note

  • When the compute engine is MaxCompute and the current table is a lakehouse table in Delta format, the number of records is not displayed.

  • The number of partition records and storage size are for reference only.

③ Lineage & Impact

• The Lineage tab displays the lineage between tables and fields.

  • Table-level lineage: Lineage sources include integration and synchronization tasks with automatically parsed lineage, SQL compute tasks, and logical table tasks, as well as custom lineage configured in compute tasks and external lineage registered through an OpenAPI. For more information, see Table-level lineage.

  • Field-level lineage: Lineage sources include integration and synchronization tasks with automatically parsed lineage (field-level lineage can be parsed only for some data sources), SQL compute tasks, and logical table tasks, as well as custom lineage configured in compute tasks and external lineage registered through an OpenAPI. For more information, see Field-level lineage.

    Note

    • For automatically parsed and manually configured lineage, the system parses the table and field lineage in the development environment when a task is submitted, and parses the lineage in the production environment when the task is published. A single task submission or publication can parse up to 100,000 lineage relationships. If this limit is exceeded, Dataphin does not record or display the lineage in the asset list.

    • Deleting a task also deletes the lineage associated with its physical table. If you delete only the physical table but not the task associated with the lineage, the lineage persists, and the corresponding table node is displayed as uncollected or deleted in the lineage graph.

    • Lineage registered through an OpenAPI takes effect immediately after the API call succeeds.

    • The system can parse table lineage when a real-time integration task is submitted or taken offline. However, after the task starts running, the system cannot parse the lineage of new or deleted tables.

    • Taking a real-time integration task offline removes its lineage. This applies only to tasks taken offline from the real-time integration task page, not to instances taken offline from the O&M page.

• Impact is divided into data table impact and integration and synchronization impact.

  • Data table impact: Displays the direct downstream tables that reference the current table, as well as the indirect downstream tables that reference the direct downstream tables. You can export the data and opt to show only direct downstream tables.

    • Export Data: Exports the data to an Excel file for business personnel to browse.

    • Show only direct downstream tables: When selected, this option displays only the direct downstream tables affected by the current table. If cleared, it shows all downstream tables up to 15 levels away.

  • Integration and synchronization impact: Displays the integration and synchronization tasks where the current table and its downstream tables are used as sources. You can export the data and opt to show the impact of only the current table.

    • Export Data: Exports the data to an Excel file for business personnel to browse.

    • Show only impact of current table: When selected, this option displays only the integration tasks that use the current table as an input. If cleared, it shows integration tasks that use the current table or any of its downstream tables as an input, up to 15 levels away.

④ Quality Overview

If you have enabled the data quality feature, this section displays a rule check summary and a list of quality monitoring rules for the current data table. Click View Report Details or View Rule Details to go to the corresponding page in the data quality module for more details.

⑤ Data Profiling

If you have enabled the data quality feature, you can configure data profiling tasks for the data table to get a quick overview of the data and assess its usability and potential risks. To enable automatic profiling, go to Governance > Metadata Center > Profiling and Analysis and enable the corresponding setting. For more information, see Create a data profiling task.

⑥ Data Preview

If sample data is available for the data table, it is displayed by default. You can also manually trigger a query to fetch the latest data. If no sample data exists, a data preview query is automatically triggered.

• Sample Data: This is displayed when both the data sampling switch and the data preview switch in the settings are turned on. The system queries sample data only for fields for which you have column-level permissions and that do not require data masking. The sample data for each field is stored and sorted independently, but the existence and correctness of the row records are not guaranteed.

• Data Preview: If you have permission to query data from the current table, you can use the data preview feature. The preview shows results only for fields for which you have SELECT permissions, including field-level and row-level permissions. You can preview up to 50 rows of data. For information on how to request permissions, see Request, renew, and release table permissions.

You can search or filter the data by field, view the details of a single row, auto-fit column widths, and expand or collapse field display names. You can also switch between row and column views. Click the sort icon next to a field to select No Sort, Ascending, or Descending order. Double-click a field value to copy it.

⑦ Output Information

Output tasks include data write tasks for this object, tasks where this table is an output based on automatic lineage parsing or custom configuration, and tasks where the node output name matches Project Name.Table Name.

The output task list is updated in near-real time, while the output details are updated on a T+1 basis.

• ① View Output Details: You can view output details for periodic tasks only. For more information, see Output Details.

• ② Go to O&M: Click the Go to O&M button to go to the task list page in the O&M Center. The page is filtered to show the current task, where you can view more information.

⑧ Instructions

You can add instructions for the data table to provide reference information for data viewers and consumers. Click Add Instructions and enter a title and content to add the instructions.

⑨ Asset Information

This section displays detailed information about the physical table, including Basic Information, Change Information, and Usage Information.

• Basic Information: Includes the table's environment, type, creation time, creator, owner, and output task.

  • Owner: The owner of the current table. You can transfer ownership to another user. In the Transfer Owner dialog box, you can choose whether to synchronously transfer ownership of the table in the development and production environments. After selecting a Recipient, click OK to complete the transfer. We recommend that you notify the recipient after the transfer. You can view transfer information on the transfer records page. For more information, see View Transfer Records.

    Note

    • A super administrator can transfer ownership of any table, while an owner can only transfer tables they own.

    • A project administrator can transfer ownership of physical tables in the projects they manage.

  • Output task: You can view the output tasks for the current table. These include data write tasks, tasks where this table is an output based on lineage parsing or configuration, and tasks where the node output name matches Project Name.Table Name. Click the name of an output task to go to its O&M details page.

    Note

    You can view output details for periodic tasks only.

• Change Information: Includes data change, last access, and DDL change information.

  • Data Change: The last time the table content was changed (corresponding to a DML operation), as collected by the Dataphin system from the compute engine. This is updated in near-real time.

  • Last Access: The last time the table was queried (corresponding to a DQL operation), based on SQL parsing within Dataphin. Access triggered by external systems is not counted. This is updated in real time.

  • DDL Change: The last time the table structure was changed (corresponding to a DDL operation), based on SQL parsing within Dataphin. Changes triggered from external systems are not counted. This is updated in real time.

• Usage Information: Includes the number of favorites, page views, and number of visits.

  • Number of favorites: The number of users who have added this table to their favorites. This is updated in real time.

  • Page views (PV): The number of page views for the current data table. The count increases by one each time the page is refreshed. This is updated in real time.

  • Number of visits: Based on SQL parsing, the count of how many times the table has been queried (corresponding to a DQL operation) in a Dataphin task. This is updated on a T+1 basis and shows the total number of visits in the last 30 days.

Area

Description

① Search and quick actions area

• Search: Search for expanded nodes on the current page by table name keywords. You can click the left/right arrows next to the search box or use the up/down arrow keys to switch between selected nodes. The system automatically locates the first matching node.

• Show only direct lineage: Click Filter > Show only direct lineage to display only the upstream and downstream tables with a direct lineage to the current table.

  Note

  • Direct lineage refers to upstream or downstream assets that are read from or written to by using statements like SELECT and INSERT.

  • Associated lineage refers to assets that are referenced by using constructs like JOIN, GROUP BY, and WHERE clauses.

  • Table-level lineage displays the corresponding indicator if any field in the table has direct lineage.

• Hide data source tables: Click Filter > Hide data source tables to hide table objects from data sources and their downstream dependencies.

• Hide uncollected nodes: Lineage is parsed when tasks are submitted and published. In some cases, if a data table is deleted outside of a Dataphin scheduled task (for example, by using an ad hoc query or directly in the compute engine), the lineage is not updated, and the table remains in the lineage graph as an uncollected node. Additionally, if lineage is registered through an API and the related objects have not yet been collected by the Dataphin system, they also appear as uncollected nodes. If you select this option, deleted or uncollected table objects are not displayed.

② Legend area

Table-level lineage supports various data tables, including physical tables, dimension logical tables, fact logical tables, summary logical tables, tag logical tables, physical views, materialized views, logical views, metatables, mirrored tables, data source tables, and .

③ Lineage graph area

This area displays the complete lineage graph. You can manually expand multiple levels of upstream or downstream dependencies and perform a fuzzy search by data table name.

• Center node: Displays the name of the current asset, marked with a Current Table label in the upper-left corner.

• Other nodes: Click a node to highlight its upstream and downstream lineage.

• Fields in a table: Hover over a node and click the drop-down arrow on the right. You can perform a fuzzy search for fields by name. Click a field name to go to the current asset's details page to view its lineage.

• Node connections: If a node has a self-dependency (for example, one field in a table depends on another field in the same table), the connection appears as a dashed line.

④ Object details area

Hover over another node to view its object details.

• Data source table: Displays the table name, object type, storage format, owning data source, and lineage source.

• Dataphin data table: Displays the table name, object type, and storage format. For logical tables and logical views, it shows the owning data section. For physical tables and physical views, it shows the owning project. It also displays the owner, storage size, lifecycle (not supported for compute engines on Hadoop clusters), description, and lineage source. You can also view lineage, view the DDL statement, and request permissions.

  • View lineage: Click to quickly view the table-level lineage for the table.

  • View DDL Statement: For physical tables, physical views, and mirrored tables, you can click to view the table's DDL statement.

  • Request Permissions: Click to quickly go to the permission request page for the table.

Area

Description

① Search and quick actions area

• Search: Search for expanded nodes on the current page by field name keywords. You can click the left/right arrows next to the search box or use the up/down arrow keys to switch between selected nodes. The system automatically locates the first matching node.

• Show only direct lineage: When selected, this option displays only the upstream and downstream fields with a direct lineage to the current field.

  Note

  • Direct lineage refers to upstream or downstream assets that are read from or written to by using statements like SELECT and INSERT. Specifically, if the calculation logic for a field is processed by using a where clause in a filter component or by using a built-in function in a field calculation component, it appears as direct lineage in the field lineage graph.

  • Associated lineage refers to assets that are referenced by using constructs like JOIN, GROUP BY, and WHERE clauses.

• Hide uncollected nodes: Lineage is parsed when tasks are submitted and published. In some cases, if a data table is deleted outside of a Dataphin scheduled task (for example, by using an ad hoc query or directly in the compute engine), the lineage is not updated, and the table remains in the lineage graph as an uncollected node. Additionally, if lineage is registered through an API and the related objects have not yet been collected by the Dataphin system, they also appear as uncollected nodes. If you select this option, deleted table objects are not displayed.

② Legend area

Field-level lineage supports fields from compute source tables and data source tables.

③ Lineage graph area

This area displays the complete lineage graph. You can manually expand multiple levels of upstream or downstream dependencies and perform a fuzzy search by field name. If a circular dependency exists, you cannot expand it further. You must start from the initial node to view its downstream dependencies.

Center node: Displays the current field and its table name, marked with a Current Field label in the upper-left corner. You can perform a fuzzy search by field keyword to switch to the lineage graph of a different field.

④ Object details area

Hover over another node to view its object details.

• Field: Displays the field name, object type, and storage format. For fields in logical tables and logical views, it shows the owning data section. For fields in physical tables and physical views, it shows the owning project. It also displays the parent table, table owner, data type, description, and lineage source. You can also view lineage and request permissions.

  • View lineage: Click to quickly view the field-level lineage for the field.

  • Request Permissions: Click to quickly go to the permission request page for the table that contains the field.

Area

Description

① Task Details

Displays the node name, task ID, data section, and owner information.

② Periodic Instance

Displays the average start time, average finish time, and average duration.

• Average start time: The average of the initial start times for all normally scheduled periodic instances in the last 15 days. This is updated on a T+1 basis.

• Average finish time: The average of the initial successful completion times for all normally scheduled periodic instances in the last 15 days. Reruns are not included in the calculation. This is updated on a T+1 basis.

• Average duration: The average of the average durations for all successfully run records of normally scheduled periodic instances in the last 15 days. For script tasks, the average duration includes reruns. For logical table tasks, the average duration does not include reruns. This is updated on a T+1 basis.

  Note

  For hourly and minutely instances, you can view run details by period. The period range is based on the latest scheduling configuration of the task. The system calculates the average run values based on historical instances with the same scheduled time.

③ Run Details

Displays the business date, status, scheduled time, start time, end time, and duration. You can also perform the View Instance and View Log operations in the Actions column.

• View Instances: Click View Instances. In the Periodic Instances list, you can view and manage tasks. For more information, see View and manage script instances or .

• View Log: View the run log for the instance.