After a data profiling task finishes, you can review the generated reports for each field data type and examine the statistical distribution of profiled fields.
Prerequisites
The Data Quality feature module must be enabled to use data profiling.
Permission description
-
Super administrators and operation administrators can view profiling reports and records for all data tables. Data table owners can only view reports and records for tables they manage.
-
Project administrators can view profiling reports and records for physical tables in their projects.
-
Section administrators can view profiling reports and records for logical tables in their sections.
-
Ordinary members can view profiling reports and records that they have permissions to access.
View data profiling results
-
On the Dataphin home page, select Administration > Asset Inventory from the top menu bar.
-
Click the Table tab. Here, you can filter by table type, including physical tables, logical tables, physical views, logical views, and materialized views.
-
In the table list, click the target table name or the Actions column's
icon to go to the object details page. -
On the Object Details page, click the Data Profile tab to view profiling results that have been successfully run and for which you have viewing permissions.
-
Profiling Record: Shows profiling records that have run successfully and have viewing permissions, including details on profiling partitions, scope, number of fields, and records profiled.
-
View Profiling Configuration: Allows you to review the configured profiling task details.
-
View Logs: Enables you to inspect the run logs for the profiling task associated with the selected record.
-
View data profiling reports
For sensitive data security, if a field has a desensitization rule applied, the original value is used for statistics, while the desensitized value is displayed in the profiling report.
For successfully executed profiling records, you can view reports that show the profiling results for each field under different profiling scenarios.
Field value distribution
Field value distribution statistics are compiled with a corresponding graph, giving you a quick overview of value distribution to support data development and pipeline applications. All data types are supported, with specific statistical indicators for each field data type.
The details of field value histograms and bar charts are as follows:
-
For numeric fields, an approximate histogram is displayed, dividing the record values into 20 intervals. A line chart illustrates the record count and average value for each interval.
-
For text, datetime, or Boolean fields, the bar chart's other values category shows values outside the Top 20 duplicates and the count of Null value records.
-
For numeric fields: Statistics include maximum (Max), minimum (Min), average (Avg), Null value count, unique value count, standard deviation, 25% quantile, median, and 75% quantile.
-
For text fields: Statistics include maximum and minimum character length, average character length, Null value count, and unique value count.
-
For datetime fields: Statistics include maximum (Max), minimum (Min), Null value count, and unique value count.
-
For Boolean fields: Statistics include the Null value count.
Null value statistics
Null value statistics help identify Null values or abnormal records in fields, which is critical for avoiding task execution errors and ensuring the accuracy of downstream data calculations. This scenario is recommended for primary key fields or fields that must not contain Null values. All data types are supported, with additional statistics for zero value records in numeric fields and empty string records in text fields.
The details of the Null value statistics donut chart are as follows:
-
For numeric fields: Statistics include the total number of records profiled, Null value count, Null value rate, zero value count, zero value rate, and other values. The donut chart provides an overview of these indicators.
-
For text fields: Statistics include the total number of records profiled, Null value count, Null value rate, empty string count, empty string rate, and other values. The donut chart provides an overview of these indicators.
-
For datetime and Boolean fields: Statistics include the total number of records profiled, Null value count, Null value rate, and other values. The donut chart provides an overview of these indicators.
If Null values or empty strings are detected in a field, consider the following recommendations:
-
If the field is a primary key (or should not contain Null values) and is numeric or text, and Null values are detected, we recommend setting up a field Null value check quality monitoring rule to prevent disruptions to downstream business processes.
-
If the field is a primary key (or should not contain Null values) and is text, and Null or empty string values are detected, we recommend setting up a field Null value check or field empty string check quality monitoring rule to prevent disruptions to downstream business processes.
Unique value statistics
Unique value statistics show the count of unique value records and the top 5 field values with the highest duplicate frequency. This scenario is recommended for primary key fields or fields with frequent duplicate occurrences. Unique value statistics are not supported for Boolean fields.
If duplicate values are found in a field, consider the following recommendation:
If the field serves as a primary key and profiling indicates duplicate values, we recommend setting up a field value uniqueness quality monitoring rule to ensure smooth downstream business processes.
View data profiling records
-
Click the View Profiling Records button to access the View Profiling Records panel.
-
In the View Profiling Records panel, you'll find the names, profiling types, statuses, and execution durations of the profiling records.
-
You can search for a specific profiling record by name or filter by profiling status and type.
-
In the target profiling record operation column, you can perform various operations.
Operation
Description
View Profiling Results
After the profiling task runs successfully, you can view the profiling report.
View Profiling Configuration
View the configuration of the profiling task. For manual profiling tasks, you can click Initiate Profiling Based On Current Configuration at the bottom to modify the settings and start a new profiling.
View Run Logs
View the run logs of the profiling task for the selected profiling record.
Stop
For manual or automatic profiling tasks that are running or waiting, you can Stop the task.
Initiate Profiling Based On Current Configuration
For manual profiling tasks, you can prefill the configuration from this task and start a new profiling. Tasks that are in progress cannot be initiated again.