Dataphin:MaxCompute Physical Table And Field Details

Section

Description

① Basic Information

Displays basic information about the table, such as its type, environment, name, tags, and description. You can also:

• Search for other assets: Quickly search for and switch to other assets to view their details.

• View Asset Details: If the object is published in the Asset Catalog, you can view its listing information on the catalog details page.

• View Production/Dev Object: Quickly switch to the details page of the corresponding object in the production or development environment.

• Tags: Displays the tags configured for the current asset. To modify tags, click Edit.

  • Each tag value can be up to 128 characters long.

  • You can configure a maximum of 20 tags for each asset.

  • The following users can modify asset tags: super administrators for all table types, table owners for tables they own, and project administrators for physical tables within their projects.

• Favorite: Click to add or remove the asset from your favorites. After adding an asset to your favorites, you can find your 30 most recent favorites in Asset List > My Footprints. You can also view all your favorited assets in the Personal Data Center. For more information, see View and manage my favorite assets.

• Go to Analyze: Click to go to the Notebook page and automatically create a corresponding Notebook task. For more information, see Create a Notebook.

• Apply for Permissions: Click to go to the permission application page for the current table. For more information, see Apply for, renew, and release table permissions.

• Report Data Quality Issue: Report data quality issues, such as unstable data generation or inaccurate content, to the asset's quality owner. This helps the owner promptly fix issues and improve the asset's availability and health. For configuration details, see Add and manage an issue list.

  This feature requires the data quality module. You can view the processing flow and results of reported issues in the data quality module's issue list.

• Generate SELECT Statement: Generates a query statement for the current table. You can choose whether to add escape characters, then copy the statement to an ad-hoc query or analysis page.

  

• View DDL Statement: View the Data Definition Language (DDL) statement for the current table in the tenant's computing engine. You can also select a Data Source Type and click Generate DDL Statement to create a DDL statement that replicates the table structure in the specified data source type. If you select Automatically add escape characters, escape characters are included to prevent conflicts with system keywords.

  

• Edit Table: Go to the Develop - Table Management page to edit the DDL information for this table. For more information, see Create an offline physical table.

• Export Fields: Exports the table's field information as a CSV file for analysis.

• View Transfer Records: Displays the 100 most recent owner transfer records for the current data table.

• View Permission List: View the permission information for the data table.

• Refresh Metadata: If a data table was created outside of Dataphin, or if a query for a new table fails due to metadata latency, click Refresh Metadata. This action fetches the latest metadata for the table and synchronizes it with Dataphin. 

② Details

Displays table, field, and partition information.

• Table Details: Displays the properties of the table, including its business segment, subject domain, project, highest security level (requires the data security feature), whether it is a partitioned table, whether it is a manually created table on the analysis platform, whether it is a lake table, lake table format, table storage mode, storage type, storage format, storage size, lifecycle, and location.

  • Project: The project to which the current table belongs. Click the project name to go to the asset details page for that project.

  • Highest Security Level: This is the highest sensitivity level among all fields in the table, providing a quick overview of its data confidentiality. Data Classification levels range from low to high: L1 to L4 (Public to Top Secret), along with custom data classifications.

  • Table Storage Mode: If the data table is a Hudi-formatted lake table, Dataphin collects this information from its compute source.

  • Storage Type: The storage type of the current table, which can be an internal table or an external table.

  • Storage Size: The actual storage size of the current table. This information is updated on a T+1 basis.

  • Lifecycle: The retention period of the current table in days.

  • Location: The storage location of the current table. This applies only to external tables that use a MaxCompute or Hive compute source. For example: hdfs://node01.exp5:8020/user/hive/warehouse.

• Field Information: Includes the field name, description, data type, associated standard (requires the data standard module), sample data (displayed only when the data sampling feature is enabled), data classification (requires the data security module), data classification level (requires the data security module), and popularity information. You can also search for, filter, and view the lineage of fields.

  • View Lineage: Click the lineage icon in the Actions column to view the field lineage centered on the specified field.

  • Search and Filter: You can search for fields by name or description, and filter fields by data classification or data classification level (requires the data security feature).

• Partition Information: Displays the partition information of the data table. Multi-level partitions are shown as a combination of each level, separated by a forward slash (/). For example: ds=20221001/pt1=a/pt2=b.

  Note

  • If the computing engine is MaxCompute and the current table is a lake table in Delta table format, the record count is not displayed.

  • The partition record count and storage size information are for reference only.

③ Lineage & Impact

• Lineage shows the lineage relationships between tables and fields.

  • Table-level Lineage sources include integration and sync tasks, data aggregation tasks, SQL compute tasks, and logical table tasks for which the system can automatically parse lineage; as well as custom lineage manually configured in compute tasks and external lineage registered through OpenAPI. For more information, see Table-level lineage.

  • Field Lineage sources include integration and synchronization tasks (field lineage can only be parsed for some data sources), data aggregation tasks, SQL compute tasks, and logical table tasks for which the system can automatically parse lineage. It also includes custom lineage manually configured in compute tasks and external lineage registered through OpenAPI. For more information, see Field lineage.

    Note

    • For automatically parsed and manually configured lineage in compute tasks, the system parses the table and field lineage in the development environment when a task is submitted. When the task is published, the system parses the table and field lineage in the production environment. A single task submission or publication can parse up to 100,000 lineage relationships. If this limit is exceeded, the lineage is not recorded and will not be displayed in the Asset List.

    • Deleting a task also deletes the lineage associated with the physical table. However, if you delete only the physical table but not the associated task, the lineage relationship persists. The table node then appears in the lineage graph as uncollected or deleted.

    • Lineage registered through OpenAPI takes effect immediately after a successful API call.

    • When a real-time integration task is submitted or taken offline, the system can parse the table lineage. However, after a real-time integration task is running, the system cannot parse lineage for newly added or deleted tables.

    • When a real-time integration task is taken offline, its lineage relationship is removed. This only affects tasks that are taken offline from the real-time integration task page. Taking an instance offline on the O&M page does not affect the lineage.

• Impact includes Data Table Impact and Integration & Sync Impact.

  • Data Table Impact: Displays the direct downstream tables that reference the current table, and the indirect downstream tables that reference those direct downstream tables. You can export the data or choose to show only direct downstream tables.

    • Export Data: Supports exporting to Excel for business users to browse.

    • Show only direct downstream tables: When selected, this option displays only the tables directly impacted by the current table. When deselected, it shows all downstream impacted tables up to a maximum of 15 levels (the direct downstream tables are considered the first level).

  • Integration & Sync Impact: Displays the integration and synchronization tasks that use the current table and its downstream tables as sources. You can export the data or choose to show only the impact of the current table.

    • Export Data: Supports exporting to Excel for business users to browse.

    • Show only the impact of the current table: When selected, this option displays only the integration tasks where the current table is used as an input. When deselected, it shows integration tasks where the current table and all its downstream impacted tables are used as inputs, up to a maximum of 15 levels (the direct downstream tables are considered the first level).

④ Quality Overview

If you have the data quality feature enabled, this section displays a rule check overview and quality monitoring rules for the current table. Click the View Report Details button or the View Rule Details button to go to the corresponding page in the data quality module for more details.

⑤ Data Profiling

If the data quality feature is enabled, you can configure a data profiling task to understand data characteristics and assess usability. To enable automatic profiling, go to Governance > Metadata Center > Profiling and Analysis and configure the settings. For more information, see Create a data profiling task.

⑥ Data Preview

The system displays sample data by default when available. You can also manually query the latest data. If no sample data exists, a data preview query runs automatically.

• Sample Data: This is displayed when the data sampling switch is on and the data preview switch in the configuration is also on. The query only returns sample data for fields to which you have column-level permissions and which do not require data masking. The system stores and sorts the sample data for each field independently, but does not guarantee that the row records exist or are correct.

• Data Preview: If you have permission to query the current table's data, you can use the data preview feature. It only returns results for fields for which you have select permissions (including field and row-level permissions). You can preview up to 50 rows of data. For information about how to apply for permissions, see Apply for, renew, and release table permissions.

You can search for or filter fields, search, view single-row details, auto-fit column widths, and transpose rows and columns. You can also click the sort icon next to a field to sort it in Ascending or Descending order, or leave it Unsorted. Double-click a field value to copy it.

⑦ Output Information

Output tasks include data write tasks for the object, tasks for which lineage is parsed or configured with the current table as the output, and tasks whose node output name is in the format ProjectName.TableName.

The output task list is updated in near real-time. The output details are updated on a T+1 basis.

• ① View Output Details: You can only view the output details of scheduled tasks. For more information, see Output details.

• ② Go to O&M: Go to the task list page in the O&M Center to filter for the current task and view more information.

⑧ Instructions

You can add instructions for a data table to provide reference information for data browsers and consumers. Click Add Instructions and enter a title and content to add the instructions.

⑨ Asset Information

Displays detailed information about the physical table, including Basic Information, Change Information, and Usage Information.

• Basic Information: Includes the table's environment, type, creation time, creator, owner, and output task.

  • Owner: The owner of the current table. You can transfer ownership of the table to another user. In the Transfer Owner dialog box, you can choose whether to simultaneously transfer the corresponding development and production environment tables. Select a Recipient and click OK to complete the transfer. We recommend that you notify the recipient after the transfer. You can view transfer information on the transfer records page. For more information, see View Transfer Records.

    Note

    • Super administrators can transfer ownership of all table types. The current table owner can transfer ownership of tables they own.

    • Project administrators can transfer ownership of physical tables within their projects.

  • Output Task: You can view the output tasks for the current table. This includes tasks that write data to the object, tasks where lineage is parsed or configured with the current table as the output, and tasks where the node output name is in the format of ProjectName.TableName. Click the name of an output task to go to the O&M details page for that data table.

    Note

    You can only view the output details of scheduled tasks.

• Change Information: Includes Data updated, Last accessed, and DDL updated.

  • Data updated: The Dataphin system collects the time of the last table content change (corresponding to a DML operation) from the computing engine. This is updated in near real-time.

  • Last accessed: This is the time of the last SELECT operation (corresponding to a DQL operation), which Dataphin parses from SQL. Access from external systems is not counted. This is updated in real-time.

  • DDL updated: The time of the last table structure change (corresponding to a DDL operation) parsed by Dataphin from SQL. Changes from external systems are not counted. This is updated in real-time.

• Usage Information: Includes Favorites, Page Views (PV), and Access count.

  • Favorites: The number of users who have favorited the current table. This is updated in real-time.

  • Page Views (PV): The number of times the current data table's page has been accessed. This count increases by one for each refresh. This is updated in real-time.

  • Access count: The number of times the table has been selected in a Dataphin task (corresponding to a DQL operation), parsed from SQL. Each select counts as one access. This is updated on a T+1 basis and shows the total access count for the last 30 days.

Section

Description

① Search and Quick Actions

• Search: Search for expanded nodes on the current page by table name keyword. You can click the left/right arrows next to the search box or use the up/down arrow keys on your keyboard to quickly switch between selected nodes. The system automatically locates the first matching node.

• Show Direct Lineage Only: Click Filter > Show Direct Lineage Only to display the upstream and downstream tables with a direct lineage relationship to the current table.

  Note

  • Direct lineage refers to upstream and downstream assets that are read from or written to using SELECT, INSERT, and similar statements.

  • Associated lineage refers to related assets referenced using JOIN, GROUP BY, WHERE clauses, and similar constructs.

  • Table-level lineage displays a direct lineage indicator if any of its fields have a direct lineage relationship.

• Hide Data Source Tables: Click Filter > Hide Data Source Tables to hide table objects from data sources and their downstream dependencies.

• Hide Uncollected Nodes: Click Filter > Hide Uncollected Nodes. Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted through a Dataphin scheduled task (for example, it is deleted through an ad-hoc query or directly in the engine), the lineage is not updated. The table will still appear in the lineage graph as an uncollected node. Additionally, for lineage registered through an API, if the related object has not yet been collected into the Dataphin system, it will also be shown as an uncollected node. When this option is selected, deleted or uncollected table objects are hidden.

② Legend

Table-level lineage supports various data table types, including physical tables, dimension logical tables, fact logical tables, aggregate logical tables, tag logical tables, physical views, materialized views, logical views, meta tables, mirrored tables, data source tables.

③ Lineage Graph Display

Displays the complete lineage graph. You can expand multiple levels of upstream or downstream nodes and search by table name.

• Center Node: Displays the name of the current asset, marked with the Current Table label in the upper-left corner.

• Other Nodes: Click a node to highlight its upstream and downstream lineage relationships.

• Fields in table: Hover over a node and click the dropdown arrow on the right. You can perform a fuzzy search by field name. Click a field name to go to the current asset's details page and view its lineage.

• Node Connection Line: If a node has a self-dependency (for example, one field in a table depends on another field in the same table), the connection is indicated by a dashed line.

④ Object Details

Hover over another node to display its details.

• Data Source Table: Displays the table's name, object type, storage format, associated data source, and lineage source.

• Dataphin Data Table: Displays the table's name, object type, and storage format. For logical tables and logical views, it shows the business segment. For physical tables and physical views, it shows the project. It also displays the owner, storage size, lifecycle (not shown for Hadoop cluster computing engines), description, and lineage source. You can also perform the View Lineage, View DDL Statement, and Apply for Permissions actions.

  • View Lineage: View the table-level lineage for the table.

  • View DDL Statement: For physical tables, physical views, and mirrored tables, view the table's DDL statement.

  • Apply for Permissions: Go to the permission application page for the table.

Section

Description

① Search and Quick Actions

• Search: Search for expanded nodes on the current page by field name keyword. You can click the left/right arrows next to the search box or use the up/down arrow keys on your keyboard to quickly switch between selected nodes. The system automatically locates the first matching node.

• Show Direct Lineage Only: When selected, this option displays only the upstream and downstream fields with a direct lineage relationship to the current field.

  Note

  • Direct lineage refers to upstream or downstream assets that are read from or written to using statements like select and insert. For example, a field shows direct lineage if its calculation logic uses a where clause in a filter or built-in functions in a calculation.

  • Associated lineage refers to assets that are referenced using methods like join, group by, or where conditions.

• Hide Uncollected Nodes: Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted through a Dataphin scheduled task (for example, it is deleted through an ad-hoc query or directly in the engine), the lineage is not updated. The table will still appear in the lineage graph as an uncollected node. Additionally, for lineage registered through an API, if the related object has not yet been collected into the Dataphin system, it will also be shown as an uncollected node. When this option is selected, deleted table objects are hidden.

② Legend

Field lineage supports fields from compute source tables and data source tables.

③ Lineage Graph Display

Displays the complete lineage graph. You can expand multiple levels of upstream or downstream nodes and search by field name. If a circular dependency exists, you cannot expand further and must view the downstream from the starting node.

Center Node: Displays the name of the current field and its table, marked with the Current Field label in the upper-left corner. You can perform a fuzzy search by field keyword to switch to the lineage graph of a different field.

④ Object Details

Hover over another node to display its details.

• Field: Displays the field's name, object type, and storage format. For fields in logical tables and logical views, it shows the business segment. For fields in physical tables and physical views, it shows the project. It also displays the parent table, table owner, data type, description, and lineage source. You can also perform the View Lineage and Apply for Permissions actions.

  • View Lineage: View the field lineage for the field.

  • Apply for Permissions: Go to the permission application page for the table containing the field.

Section

Description

① Task Details

Displays the Node Name, Task ID, Business Segment, and Owner.

② Scheduled Instance

Displays the Average Start Time, Average Output Time, and Average Duration.

• Average Start Time: The average of the initial run start times for all normally scheduled instances over the last 15 days. Updated on a T+1 basis.

• Average Output Time: This is the average end time of the first successful run for all normally scheduled instances over the last 15 days. Reruns are not included. This value updates on a T+1 basis.

• Average Duration: The average duration across all successful runs of normally scheduled instances over the last 15 days. For script tasks, reruns are included in the calculation. For logical table tasks, reruns are excluded. Updated on a T+1 basis.

  Note

  For hourly and minute-based instances, you can view run details by period. The system calculates the average run metrics for historical instances with the same scheduled time, based on the cycle intervals defined by the latest scheduling configuration of the task.

③ Run Details

Displays the Business Date, Status, Scheduled Time, Start Time, End Time, and Duration. You can also perform the View Instance and View Log actions in the Actions column.

• View Instance: Click the View Instance button to view and manage tasks in the Scheduled Instance list. For more information, see View and manage script instances.

• View Log: View the detailed run log for the task instance.