Dataphin:Data source table details

Area

Description

① Basic information

Displays information about the metadata, such as its name, environment, tags, and description. You can also perform the following operations:

For data source tables from an Elasticsearch data source, you can search for metadata, view asset details, view production or development objects, modify tags, add items to favorites, report quality issues, request data source permissions, and export fields. The export feature is supported only for table objects.

• If the metadata is deleted from the source system, its name is grayed out. You can click the icon next to the name to go to the Metadata Checklist page and delete the metadata that is no longer used.

• Search Metadata: In the search box in the upper-right corner, enter keywords such as a table name to search for data source tables. You can also switch to view the details of other assets.

• View Asset Details: If the current object is listed in the Asset Directory, you can click this button to go to the directory details page and view the listing information.

• View Production/Development Object: In the upper-right corner, click View Production/Development Object to switch to the details page of the object in the corresponding production or development environment.

• Tag: Displays the tags that are configured for the current asset. To modify tags, click Edit.

  • Each tag value can be up to 128 characters long.

  • You can configure a maximum of 20 tags for each asset object.

• Favorite: Click to add to or remove from favorites. After adding to favorites, you can view the 30 most recently favorited assets in Asset Checklist > My Footprint for easy subsequent queries. You can also view all favorited assets in the Personal Data Center. For more information, see View and manage my favorited assets.

• Go Analysis: Click to go to the Notebook page to automatically create a corresponding Notebook node. You can perform visual analytics on this asset. For more information, see Create a Notebook.

  Note

  For some data source tables, after you request consumption permissions, you can create Notebook tasks. For information about the supported data sources, see Operations supported for different types of assets.

• Request Permission: Click to go to the permission request page for the current data table. For more information, see Request, renew, and return table permission.

• Feedback quality issues: If the metadata has been deleted, this operation is not supported. You can use this feature to report quality issues encountered during asset usage (such as unstable data output or inaccurate data content) to the current quality owner, so that the relevant owner can rectify the issues in a timely manner, thereby improving asset availability and health. For configuration details, see Add and manage issue checklist.

  You must enable the Data Quality module to use this feature. You can report quality issues only for data source tables in the production environment. You can then view the processing flow and results of the reported issues in the issue checklist of the Data Quality module.

• Generate SELECT Statement: Click to generate a query statement for the current table. You can choose whether to add escape characters. You can copy the query statement and run it on the ad hoc query or analysis page.

• View DDL Statement: In the upper-right corner, click More and select View DDL Statement to view the Data Definition Language (DDL) statement for the current metadata in its data source. You can also select a Data Source Type and click Generate DDL Statement. The system generates a DDL statement to create a table that has the same structure as the current metadata in the specified type of data source. If you select Automatically Add Escape Characters, the generated DDL statement automatically includes the appropriate escape characters for the selected source type. This reduces issues such as incorrect escaping of system keywords.

• Request Data Source Permission: Click More in the upper-right corner and select Request Data Source Permission to go to the permission request page for the current data source. For more information, see Request, renew, and return data source permission.

• Export Fields: In the upper-right corner, click More and select Export Fields to export the field information of the metadata. The information is exported in CSV format for quick analysis and use by developers or business analysts.

② Detail information

Displays the attribute and field information of tables and views.

• Detail: Displays the business and technical attributes of the metadata.

  • Business attributes: Includes source system information.

  • Technical attributes: Includes the source data source, number of table rows, highest sensitivity level (requires the Data Security feature), whether the table is a lakehouse table, lakehouse table format, table storage mode, storage class (internal or foreign table), storage size, and location information.

    • Table storage mode: If the data table is a lakehouse table and its format is Hudi, this information can be collected from the table's data source. This information cannot be collected for StarRocks, SelectDB, or Doris data sources.

    • Location: The storage address of the current table. For example: hdfs://node01.exp5:8020/user/hive/warehouse.

      Note

      • The technical attributes of a view only include the source data source information.

      • Storage class information is not available for MySQL or Microsoft SQL Server data sources.

      • For Oracle tables, the storage size is an estimate based on the number of rows and the average space used per row in bytes. It does not include statistics for indexes, unused space, or fragmentation. Therefore, the estimated value is lower than the actual space occupied.

      • If the data source table is a partitioned table, the row count is for reference only. The system does not currently support collecting the row count of the main table in a partitioned table.

      • For Hive and Greenplum data sources, you can view location information if the storage class is a foreign table.

      • For Amazon Redshift data sources, you can only view the source data source and storage class information.

      • For Elasticsearch data sources, if the object is a table (index), you can only view the document count and storage size.

• Field Information: Displays the field information for tables and views. This includes the field name, data type, description, whether the field can be null, default value, and data classification or sensitivity level (requires the Data Security module). For Elasticsearch data sources, if the object is a table (index), you can view the field name and data type.

  Search and filter: You can search for fields by name or description. You can also filter for specific fields by data classification or sensitivity level (requires Data Security).

• Partition Info: View the partition information of the data table. Multi-level partitions are displayed as a combination of all partition levels, separated by forward slashes (/). For example: ds=20221001/pt1=a/pt2=b.

  Note

  • The partition record count and storage size are for reference only.

  • For Elasticsearch data sources, you cannot view partition information.

③ Lineage & impact

• Lineage: Displays the data lineage between tables and fields.

  • Table-level lineage sources include integrated sync tasks, custom lineage compute tasks, SQL compute tasks that the system can automatically parse, and logical table tasks. For more information, see Table-level lineage.

  • Field lineage sources include custom lineage compute tasks, and SQL compute tasks and logical table tasks that the system can automatically parse. For more information, see Field lineage.

    Note

    The system parses table and field lineage in the developer environment when a task is submitted. It parses lineage in the production environment when the task is published. A single task submission or publication can parse a maximum of 100,000 lineage relationships. If this limit is exceeded, the lineage is not recorded and cannot be displayed in the Asset Directory.

• Impact: Impact is divided into data table impact and sync impact.

  • Data table impact: Displays the direct child tables that reference the current table and the indirect descendant tables that reference the direct child tables. You can export the data or display only the direct child tables.

    • Export Data: Export the data to an Excel file for business analysts to browse.

    • Display Only Direct Descendant Tables: Select this option to display only the downstream tables that are directly affected by the current table. Deselect this option to display all affected downstream tables, up to 15 levels. The direct downstream tables of the current table are considered the first level.

  • Sync impact: Displays the sync tasks where the current table and its descendant tables are used as source tables. You can export the data or display only the impact of the current table.

    • Export Data: Export the data to an Excel file for business analysts to browse.

    • Display Only Current Table Impact: Select this option to display only the sync tasks where the current table is used as an input table. Deselect this option to display the sync tasks where the current table and all its affected downstream tables are used as input tables, up to 15 levels. The direct downstream tables of the current table are considered the first level.

④ Quality overview

You need to enable the Data Quality feature to view the rule verification overview and quality monitoring rule list of the current data table. You can click the View Report Details button or the View Rule Details button to quickly jump to the corresponding page of the Data Quality module for more details. You can view the quality overview only for data tables that support data quality monitoring. For a list of supported data sources, see Data Sources Supported by Dataphin.

You can create quality rules and view the quality overview only for data source tables in the production environment.

⑤ Data exploration

If you have enabled the Data Quality feature, you can initiate and view data explorations for data source tables that support this feature to quickly understand the data overview and assess the data's availability and potential threats in advance. To enable automatic exploration, you can enable the corresponding configuration in Administration > Metadata Center > Exploration and Analysis. For details about how to configure exploration tasks, see Create a data exploration task. For data sources that support data exploration, see Exploration partitions and ranges supported by different data sources.

⑥ Data preview

If you have permission to query data from the current table, you can use the data preview feature to query only the results corresponding to the fields for which you have SELECT permission. You can preview the first 50 data entries. For more information about how to request query permission, see Request, renew, and return table permission.

After you filter the data, you can search or filter by field, view the details of a single row, automatically adjust column widths, and transpose rows and columns. You can also click the sort icon next to a field to sort the data. The options are No Sort, Ascending, and Descending. Double-click a field value to copy it.

⑦ Asset information

Displays the basic and usage information of the data source table.

• Basic Information: Includes the environment, table type, last collection time, and collection task of the data source table.

  • Last Collection Time: The time when the last successful run of the collection task for the current data source table was completed. The format is YYYY-MM-DD hh:mm:ss.

  • Collection Task: The collection task information for the current data source table.

• Usage Information: Includes the number of favorites and page views for the data source table.

  • Number Of Favorites: The number of users who have added the current data source table to their favorites. This value is updated in real time.

  • Page Views: The number of page views (PV) for the current data source table. The count increases by one each time the page is refreshed. This value is updated in real time.

Ordinal number

Description

① Quick actions

• Search: Search for expanded nodes on the current page by table name. Click the left or right arrow next to the search box to switch between the matched nodes. The system locates the first matched node by default.

• Display Only Direct Lineage: Click Filter and select Display Only Direct Lineage to display the upstream and downstream tables that have a direct lineage relationship with the current table.

  Note

  • Direct lineage refers to upstream and downstream assets that are read from or written to using statements such as SELECT and INSERT.

  • Associated lineage refers to related assets that are referenced using methods such as JOIN, GROUP BY, and WHERE conditions.

  • Table-level lineage displays the corresponding identifier if it contains any fields with direct lineage.

• Hide Data Source Table: Click Filter and select Hide Data Source Table to hide table objects from the data source and their downstream lineage.

• Hide Deleted Tables: Click Filter and select Hide Deleted Tables. Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted by a Dataphin scheduling task (for example, it is deleted by an ad hoc query or directly in the DPI engine), the lineage is not updated and is still displayed in the lineage graph. If you select this option, deleted table objects are not displayed.

② Legend

Table-level lineage supports the following data tables: Physical Table, Logical Dimension Table, Logical Fact Table, Logical Summary Table, Logical Tag Table, View, Materialized View, Logical View, Meta Table, Mirror Table, and Datasource Table.

③ Lineage graph display

Displays the complete lineage graph. You can manually expand multiple levels of upstream or downstream nodes and perform a fuzzy search by table name. If a circular dependency exists, you cannot expand the lineage further. You must view the downstream lineage from the start node.

• Central node: Displays the name of the current asset. The node is marked with Current Table in the upper-left corner.

• Fields in the table: Hover over a node and click the drop-down arrow on the right to perform a fuzzy search by field name. Click a field name to go to the asset details page and view the lineage relationship.

④ Object details

Hover over a table to view its details.

For a data source table, the details include its Name, Object Type, Data Source Type, Data Source, and Lineage Source. You can also perform the following operations: View Lineage, View DDL, and Request Permission.

• If the metadata is deleted from the source system, you can click the icon next to the name to go to the Metadata Checklist page and delete the metadata that is no longer used.

• View Lineage: Click to view the table-level lineage of the table.

• View DDL Statement: You can click this button to view the DDL statement for a physical table, physical view, or mirror table.

• Request Permission: This operation is not supported if the metadata is deleted. For metadata with a Normal status, you can click this button to go to the table's permission request page.

Ordinal number

Description

① Quick actions

• Search: Search for expanded nodes on the current page by field name. Click the left or right arrow next to the search box to switch between the matched nodes. The system locates the first matched node by default.

• Display Only Direct Lineage: Select this option to display only the upstream and downstream fields that have a direct lineage relationship with the current field.

  Note

  • Direct lineage refers to upstream and downstream assets that are read from or written to using statements such as SELECT and INSERT. In a field lineage graph, if the calculation logic of a field is processed using a where condition in a filter component or a built-in function in a field calculation component, the lineage is displayed as direct lineage.

  • Associated lineage refers to related assets that are referenced using methods such as JOIN, GROUP BY, and WHERE conditions.

• Hide Deleted Fields: Lineage is parsed when tasks are submitted and published. In some cases, if a data table is not deleted by a Dataphin scheduling task (for example, it is deleted by an ad hoc query or directly in the DPI engine), the lineage is not updated and is still displayed in the lineage graph. If you select this option, deleted table objects are not displayed.

② Legend

Field lineage supports the following fields: Compute Source Table Field and Data Source Table Field.

③ Lineage graph display

Displays the complete lineage graph. You can manually expand multiple levels of upstream or downstream nodes and perform a fuzzy search by field name. If a circular dependency exists, you cannot expand the lineage further. You must view the downstream lineage from the start node.

Central node: Displays the current field and its table name. The node is marked with Current Field in the upper-left corner. You can perform a fuzzy search by field keyword to switch the view to the lineage graph of a different field.

④ Object details

Hover over a field to view its details. The details include the Name, Object Type, Data Source Type, Table, Owner, Data Type, Description, and Lineage Source. You can also perform the View Lineage operation.

• Table: If a collection task is configured for the data source table, click the table name to go to the details page of the data source to which the table belongs.

• Request Permission: For metadata with a Normal status, you can click this button to go to the table's permission request page.

• View Lineage: Click to view the lineage information of the field as the central field.