Dataphin:MaxCompute physical table and field details

Area

Description

① Summary information

Displays information about the table, such as its type, environment, name, tags, and description. You can also perform the following operations:

• Search for other asset objects: Quickly search for and switch to other assets to view their details.

• View Asset Details: If the current object is listed in the Asset Directory, you can go to the directory details page to view the listing information.

• View Production/Development Objects: Quickly switch to the object details in the corresponding production or developer environment.

• Tag: Displays the tag values configured for the current asset. To modify the tags, click Edit:

  • Each tag value can be up to 128 characters long.

  • You can configure a maximum of 20 tag values for each asset object.

  • A super administrator can modify asset tags for all table types. A table owner can modify asset tags for the tables they own. A project administrator can modify asset tags for physical tables in the projects they manage.

• Favorite: Click to add the asset to or remove it from your favorites. After you add an asset to your favorites, you can view the 30 most recently added assets in Asset Checklist > My Footprint for quick access. You can also view all your favorite assets in the Personal Data Center. For more information, see View and manage my favorite assets.

• Go Analysis: Go to the Notebook page to automatically create a Notebook. For more information, see Create a Notebook.

• Request Permission: Go to the permission request page for the current table. For more information, see Request, renew, and return table permissions.

• Feedback quality issues: Use this feature to provide feedback to the quality owner about issues that you encounter when you use the asset, such as unstable data output or inaccurate data content. This notifies the relevant owner to make corrections promptly, which helps improve asset availability and health. For more information about the configuration, see Add and manage an issue checklist.

  You must enable the Data Quality module to use the feedback feature. You can then view the processing flow and results of the feedback in the issue checklist of the Data Quality module.

• Generate select statement: Generate a search statement for the current table. You can select whether to add escape characters. You can copy the search statement to the ad hoc query or analysis page to query data.

  

• View DDL Statement: View the Data Definition Language (DDL) statement that corresponds to the current data table in the tenant compute engine. You can also select a Data Source Type and click Generate DDL Statement. The system generates a DDL statement to create a table that has the same structure as the current table in the specified type of data source. If you select Automatically Add Escape Characters, the generated DDL statement automatically adds the corresponding escape characters based on the selected source type. This reduces issues such as incorrect escaping of system keywords.

  

• Edit Table: Go to the Developer - Table management page to edit the DDL information of the table. For more information, see Create an offline physical table.

• Export Fields: Export the field information of the table in the CSV format. This helps other developers or business analysts quickly analyze and use the information.

• View Transfer-out Records: Displays the 100 most recent owner change records for the current data table.

• View Permission List: View the permission information of the data table.

• Refresh Metadata: If a data table was not created on the Dataphin platform, or if a query for a new table returns no results due to a data retrieval delay, click Refresh Metadata. This obtains the latest metadata and refreshes the metadata of the specified data table to the Dataphin system. 

② Detail information

Displays information about the table, fields, and partitions.

• Detail: Displays the properties of the table. The properties include the data category, subject area, project, highest sensitivity level (requires the purchase of the data security feature), whether the table is a partitioned table, whether the table is manually created on the analysis platform, whether the table is a lakehouse table, lakehouse table format, table storage mode, storage class, storage format, storage size, lifecycle, and location.

  • Project: The project to which the current table belongs. Click the project name to go to the asset details page of the project.

  • Highest sensitivity level: The highest sensitivity level among the fields in the current table. This helps you quickly understand the data confidentiality of the table. The Data Classification levels from low to high are L1, L2, L3, and L4 (from public to top secret), along with custom data classifications.

  • Table storage mode: If the data table is a lakehouse table and its format is Hudi, this information can be collected from the source table, which is the compute source.

  • Storage type: The storage class of the current table. Valid values: internal table and foreign table.

  • Storage size: The actual storage size of the current table. The value is updated at T+1.

  • Lifecycle: The retention period of the current table in days.

  • Location: For foreign tables whose compute source is MaxCompute or Hive, you can view the storage address of the current table. Example: hdfs://node01.exp5:8020/user/hive/warehouse.

• Field Information: Includes the field name, description, data type, associated standard (requires the purchase of the Data Standard module), sample data (displayed only if you enable the data sampling feature), data classification (requires the purchase of the data security module), data classification (requires the purchase of the data security module), and heat information. You can also search for, filter, and view the lineage of fields.

  • View lineage relationship: Click the lineage relationship icon in the Actions column to view the field lineage centered on the specified field.

  • Search and filter: You can search for fields by name or description. You can also filter fields by data classification and data sensitivity level. Data security must be enabled.

• Partition Info: View the partition information of the data table. Multi-level partitions are displayed as a combination of partitions at each level, separated by forward slashes (/). For example: ds=20221001/pt1=a/pt2=b.

  Note

  • When the compute engine is MaxCompute and the current table is a lakehouse table in the Delta table format, the number of records is not displayed.

  • The number of partition records and storage size are for reference only.

③ Lineage & impact

• Lineage displays the data lineage between tables and fields.

  • Table-level lineage sources include sync tasks, SQL compute tasks, and logical table tasks for which lineage can be automatically parsed by the system, custom lineage that is manually configured for compute tasks, and external lineage that is registered using OpenAPI. For more information, see Table-level lineage.

  • Field lineage sources include sync tasks (field-level lineage can be parsed for only some data sources), SQL compute tasks, and logical table tasks for which lineage can be automatically parsed by the system, custom lineage that is manually configured for compute tasks, and external lineage that is registered using OpenAPI. For more information, see Field-level lineage.

    Note

    • For lineage that is automatically parsed by the system and lineage that is manually configured in compute tasks, the system parses the table and field lineage in the developer environment when a task is submitted, and parses the table and field lineage in the production environment when the task is published. A single task submission or publication supports parsing of up to 100,000 lineage relationships. If this limit is exceeded, the relationships are not recorded and cannot be displayed in the Asset Checklist.

    • Deleting a task also deletes the lineage associated with the physical table. If you delete only the physical table but not the task associated with the lineage, the lineage still exists. The corresponding table node in the lineage graph is displayed as uncollected or deleted.

    • For lineage registered using OpenAPI, the lineage takes effect immediately after the API call is successful.

    • When a real-time sync task is submitted or unpublished, the system can parse the table lineage. After the real-time sync task runs, the system cannot parse the lineage of new or deleted tables.

    • After a real-time sync task is unpublished, the lineage is deleted. This affects only tasks that are unpublished on the real-time sync task page. Instances that are taken offline on the O&M page are not affected.

• Impact refers to the effects on data tables and integration synchronization.

  • Data table impact: Displays the direct descendant tables that reference the current table and the indirect descendant tables that reference the direct descendant tables. You can export data and display only direct descendant tables.

    • Export Data: Export data to an Excel file for business analysts to view.

    • Display Only Direct Descendant Tables: If you select this option, only the descendant tables that are directly affected by the current table are displayed. If you clear this option, all affected descendant tables are displayed, up to a maximum of 15 levels. The direct descendants of the current table are at the first level.

  • Sync task impact: Displays the sync tasks that use the current table and its descendant tables as source tables. You can export data and display only the impact on the current table.

    • Export Data: Export data to an Excel file for business analysts to view.

    • Display Only Current Table Impact: If you select this option, only the sync tasks that use the current table as an input table are displayed. If you clear this option, the sync tasks that use the current table and all affected descendant tables as input tables are displayed, up to a maximum of 15 levels. The direct descendants of the current table are at the first level.

④ Quality overview

If you enable the Data Quality feature, an overview of rule validation and a list of quality monitoring rules for the current data table are displayed. Click View Report Details or View Rule Details to go to the corresponding page in the Data Quality module for more details.

⑤ Data exploration

If you enable the Data Quality feature, you can configure data exploration tasks for the data table to quickly understand the data profile and assess data availability and potential threats in advance. If you want to enable automatic exploration, you can enable the corresponding configuration in Administration > Metadata Center > Exploration and Analysis. For more information, see Create a data exploration task.

⑥ Data preview

If sample data exists for the data table, the sample data is displayed by default. You can also manually trigger a query to obtain the latest data. If no sample data exists, a data preview query is automatically triggered.

• Sample data: This feature is available when data sampling is enabled and the data preview switch in the usage configuration is turned on. You can query only the sample data for fields for which you have column-level permissions and that do not require data masking. The system stores and sorts the sample data for each field independently, but does not guarantee the existence or correctness of row records.

• Data preview: If you have permissions to query the current table, you can use the data preview feature. You can query only the results for fields for which you have permissions for SELECT statements, including field-level and row-level permissions. You can preview the first 50 data records. For more information about how to request permissions, see Request, renew, and return table permissions.

You can search or filter the data by field, view the details of a single row, adjust the column width, and transpose rows and columns. You can also click the sort icon next to a field to select No Sort, Ascending, or Descending. Double-click a field value to copy it.

⑦ Output information

Output tasks include data write tasks for the object, tasks for which lineage is automatically parsed or custom-configured with the current table as the output table, and tasks whose output name is in the format of `Project name.Table name`.

The output task list is updated in near real-time. The output details are updated at T+1.

• ① View Output Details: You can view the output details of only auto triggered tasks. For more information, see Output details.

• ② Go To O&M: Click Go To O&M to go to the task list page in the Operation Center. The current task is automatically filtered. You can view more information on the page.

⑧ Usage instructions

You can add usage instructions for the data table to provide information for data viewers and consumers. Click Add Usage Instructions, and enter a title and content.

⑨ Asset information

Displays detailed information about the physical table, such as Basic Information, Change Information, and Usage Information.

• Basic information: Includes the environment, table type, creation time, creator, owner, and output tasks of the table.

  • Owner: The owner of the current table. You can change the owner of the current table to another user. In the Change Owner dialog box, you can select whether to synchronize the owner change for the table in the developer or production environment. Select a Recipient and click OK to immediately transfer ownership. We recommend that you notify the recipient promptly after the transfer. You can view the transfer information on the transfer records page. For more information, see View Transfer-out Records.

    Note

    • A super administrator can change the owner of all table types. A table owner can change the ownership of the tables they own.

    • A project administrator can change the owner of physical tables in the projects they manage.

  • Output Task: You can view the output tasks of the current table. These include data write tasks for the object, tasks for which lineage is parsed or configured with the current table as the output table, and tasks whose output name is in the format of `Project name.Table name`. Click the name of an output task to go to the O&M details page of the data table.

    Note

    You can view the output details of only auto triggered tasks.

• Change information: Includes data changes, recent access, and DDL changes.

  • Data Updated At: The time of the last table content change (corresponding to DML operations) that Dataphin collects from the compute engine. The value is updated in near real-time.

  • Last Accessed At: The time of the last SELECT operation (corresponding to Data Query Language (DQL) operations) that is parsed by Dataphin based on SQL. Access triggered by external systems is not counted. The value is updated in real time.

  • Last Ddl Time: The time of the last table schema evolution (corresponding to DDL operations) that is parsed by Dataphin based on SQL. Changes triggered by external systems are not counted. The value is updated in real time.

• Usage information: Includes the number of favorites, page views, and visits.

  • Number Of Favorites: The number of users who have added the current table to their favorites. The value is updated in real time.

  • Page Views: The number of page views (PVs) for the current data table. The value increases by one each time the page is refreshed. The value is updated in real time.

  • Visits: Based on SQL parsing, each time the table is selected in a Dataphin task (corresponding to a DQL operation), it is counted as one visit. The value is updated at T+1. The total number of visits in the last 30 days is displayed.

Area

Description

①Search and quick operation area

• Search: Search for expanded nodes on the current page by table name keyword. You can click the left or right arrow next to the search box or use the arrow keys on your keyboard to switch between selected nodes. The system automatically locates the first matching node.

• Display Only Direct Lineage: Click Filter - Display only direct lineage to show the upstream and downstream tables that have a direct lineage with the current table.

  Note

  • Direct lineage refers to upstream and downstream assets that are read from or written to using statements such as SELECT and INSERT.

  • Associated lineage refers to related assets that are referenced using methods such as JOIN, GROUP BY, and WHERE conditions.

  • Table-level lineage displays the corresponding identifier as long as it contains fields with direct lineage.

• Hide Source Tables: Click Filter - Hide source tables to hide table objects from data sources and their downstream dependencies.

• Hide Uncollected Nodes: Click Filter - Hide Uncollected Nodes. Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted by a Dataphin scheduling task, for example, it is deleted through an ad hoc query or directly in the DPI engine, the lineage is not updated. The table is still displayed in the lineage graph as an uncollected node. In addition, for lineage registered using an API, if the related object has not been collected into the Dataphin system, it is also displayed as an uncollected node. After you select this option, deleted or uncollected table objects are not displayed.

②Legend area

The data tables supported by table-level lineage include Physical Table, Logical Dimension Table, Logical Fact Table, Logical Summary Table, logical tag table, View, Materialized View, Logical View, Meta Table, Mirror Table, and Datasource Table.

③Lineage graph display area

Displays the complete event chain diagram. You can manually expand multiple levels of upstream or downstream dependencies. You can perform a fuzzy search by data table name keyword.

• Central node: Displays the name of the current asset and is marked with Current Table in the upper-left corner.

• Other nodes: Click a node to highlight its upstream and downstream lineage.

• Fields within the table: Hover over a node and click the drop-down arrow on the right. You can perform a fuzzy search by field name. Click a field name to go to the current asset details page to view the lineage.

• Node connection: If a node has a self-dependency, such as one field in the table depending on another, it is indicated by a dashed line.

④Object details area

Hover over a table to view its details.

The data source table lists the Name, Object Type, Storage Format, Data Source, and Lineage Source.

Dataphin data table: Displays the table's Name, File Format, Storage File Format, the Subject Area of logical tables and logical views or the Project of physical tables and physical views, Owner, Storage Size, Lifecycle (this information is not displayed for compute engines on Hadoop clusters), Description, and Lineage Source. You can also perform the View Lineage, View DDL, and Request Permission operations.

• View Lineage: Quickly view the table-level lineage of the table.

• View DDL Statement: If the data table is a physical table, physical view, or mirror table, you can click to view its DDL statement.

• Request Permission: Go to the permission request page for the table.

Area

Description

①Search and quick operation area

• Search: Search for expanded nodes on the current page by field name keyword. You can click the left or right arrow next to the search box to switch between matching nodes. The system automatically locates the first matching node.

• Display Only Direct Lineage: If you select this option, only the upstream and downstream fields that have a direct lineage with the current field are displayed.

  Note

  • Direct lineage refers to upstream and downstream assets that are read from or written to using statements such as SELECT and INSERT. Specifically, if you process the calculation logic of a field using a where condition in a filter widget, or use a built-in function in a field calculation widget, it is displayed as direct lineage in the field lineage graph.

  • Associated lineage refers to related assets that are referenced using methods such as JOIN, GROUP BY, and WHERE conditions.

• Hide Uncollected Nodes: Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted by a Dataphin scheduling task, for example, it is deleted in an ad hoc query or directly in the DPI engine, the lineage is not updated. The table is still displayed in the lineage graph as an uncollected node. In addition, for lineage registered using an API, if the related object has not been collected into the Dataphin system, it is also displayed as an uncollected node. After you select this option, deleted table objects are not displayed.

②Legend area

The fields supported by field lineage include Compute Source Table Fields and Data Source Table Fields.

③Lineage graph display area

Displays the complete event chain diagram. You can manually expand multiple levels of upstream or downstream dependencies. You can perform a fuzzy search by field name keyword. If a circular dependency exists, you cannot expand it further. You must view the downstream dependencies from the starting node.

Central node: Displays the current field and its table name, and is marked with Current Field in the upper-left corner. You can perform a fuzzy search by field keyword to switch to and view the lineage graphs of different fields.

④Object details area

Hover over a field to display its Name, Object Type, Storage Format, the Board for the logical table or logical view, the Project for the physical table or physical view, the Table, the Table Owner, the Data Type, the Description, and Lineage Source. You can also View Lineage and Request Permission.

• View Lineage: Quickly view the field-level lineage of the field.

• Request Permission: Go to the permission request page for the table where the field is located.

Area

Description

①Task details

Displays the Node Name, Task ID, Subject Area, and Owner.

②Recurring instance

Displays the Average Start Time, Average Output Time, and Average Running Duration.

• Average Start Time: The average start time of the first run of all normally scheduled recurring instances in the last 15 days. The value is updated at T+1.

• Average Output Time: The average end time of the first successful run of all normally scheduled recurring instances in the last 15 days. Reruns are not included in the statistics. The value is updated at T+1.

• Average Running Duration: The average of the average running durations of all successful runs of all normally scheduled recurring instances in the last 15 days. The average running duration of script tasks includes reruns, while the average running duration of logical table tasks does not. The value is updated at T+1.

  Note

  For hourly and minutely instances, you can view running details by epoch. The epoch is determined by the latest scheduling configuration of the current task. The average value is calculated based on historical instances with the same scheduled time.

③Running details

Displays the Data Timestamp, Status, Scheduled Time, Start Time, End Time, and Running Duration. You can also perform the View Instance and View Log operations in the Actions column.

• View Instance: Click View Instance. In the Recurring Instance list, you can view and manage tasks. For more information, see View and manage script instances.

• View Log: View the operational log details of the task instance.