Dataphin lets you create quality rules to validate data tables, which improves the efficiency of monitoring table quality. This topic describes how to configure quality rules for enterprise-wide data tables.
Prerequisites
You must add a monitored object before you can configure a quality rule. For more information, see Add a Monitored Object.
Permissions
Super administrators, quality administrators, and custom global roles with the Quality Rule - Manage permission can configure scheduling, alerting, exception archive tables, and scoring weights for quality rules.
A quality owner can configure scheduling, alerting, exception archiving tables, and scoring weights for quality rules under their assigned monitored objects.
Quality owners and regular users require additional sync-read permissions for the data source containing the enterprise-wide data table. To request this permission, see Request Data Source Permissions.
Supported operations vary by object type. For more information, see Quality Rule Operation Permissions.
Validation Rule Overview
When a data table triggers a quality soft rule, Dataphin sends an alert, helping you detect and fix issues quickly. When it triggers a quality strong rule, Dataphin stops all downstream tasks that depend on the table, preventing dirty data from flowing downstream. Dataphin also sends an alert, enabling quick detection and resolution of issues.
Difference Between Trial Run and Execution
Trial runs and executions differ in their operation and the results they display. A trial run simulates a one-time execution of a quality rule. You can use it to test whether the rule is correct and works as expected. Trial run results do not appear in quality reports. Execution runs the quality rule at scheduled times. Execution results appear in quality reports, allowing you to review and analyze them.
Configure a Quality Rule
In the top menu bar on the Dataphin homepage, choose Administration > Data Quality.
In the navigation pane on the left, click Quality Rule. On the Enterprise-Wide Data Table page, click the name of the target object. Go to the Quality Rule Details page to configure the rule.
On the Quality Rule Details page, click Create Quality Rule.
In the Create Quality Rule dialog box, configure the parameters.
Parameter
Description
Basic Information
Rule Name
Enter a custom name for the quality rule. Maximum length: 256 characters.
Rule Strength
Supports Soft Rule and Strong Rule.
Weak Rule: If you select Weak Rule, Dataphin sends an alert when validation fails. It does not block downstream task nodes.
Strong Rule: If you select Strong Rule, Dataphin sends an alert when validation fails. If downstream tasks exist (for example, code-check scheduling or task-triggered scheduling), Dataphin blocks those tasks to prevent data pollution. If no downstream tasks exist (for example, periodic quality scheduling), Dataphin only sends an alert.
Description
Enter a custom description for the quality rule. Maximum length: 128 characters.
Configuration Method
Template-Based Creation: Use built-in system templates or custom business templates to create quality rules quickly.
System Template: Built-in parameters are configurable. Best for general-purpose rule creation.
Custom Template: Parameters are preconfigured. Best for rules with business logic.
SQL: Use SQL to define flexible, custom quality monitoring rules. Best for complex scenarios.
Rule Template
Select a template from the drop-down list: Completeness, Uniqueness, Timeliness, Validity, Consistency, Stability, or Custom SQL.
Completeness: Includes Null Field Check and Empty String Field Check.
Uniqueness: Includes Field Uniqueness Check, Field Group Count Check, and Field Duplicate Value Count Check.
Timeliness: Includes Time Function Comparison, Single-Table Time Field Comparison, and Two-Table Time Field Comparison.
Validity: Includes Field Format Check, Field Length Check, Field Range Check, Lookup Table Reference Check, and Data Standard Lookup Table Reference Check (requires Data Standard module).
Consistency: Includes Single-Table Field Value Consistency Check, Single-Table Field Statistical Value Consistency Check, Single-Field Business Logic Consistency Check, Two-Table Field Value Consistency Check, Two-Table Field Statistical Value Consistency Check, Two-Table Field Business Logic Consistency Check, and Cross-Source Two-Table Field Statistical Value Consistency Check.
Stability: Includes Table Stability Check, Table Volatility Check, Field Stability Check, and Field Volatility Check.
Custom SQL: Includes Custom Statistical Metric Check and Custom Data Detail Check.
For more information, see Template Types.
Rule Type
This depends on the selected template. It is a basic property used for description and filtering.
Monitoring Granularity
Available only when you select Custom SQL. Choose Entire Table or specify a Field to monitor.
Template Configuration
Template Information
Shows configuration details when you select a quality rule template. To change these settings, go to Quality Rule Templates.
Rule Configuration
Rule Configuration
Options vary based on your selected rule template. For more information, see Data Table Parameter Settings.
Validation Table Data Filtering: Disabled by default. When enabled, set filter conditions, partition filters, or standard data filters for the validation table. These filters append directly to the validation SQL. If your validation table needs partition filtering, configure the partition filter expression in the scheduling configuration. After configuration, quality reports use the validation partition as the smallest viewing unit.
When you select Consistency/Two-Table Field Statistical Value Consistency Check or Consistency/Cross-Source Two-Table Field Statistical Value Consistency Check, you can enable Comparison Table Data Filtering. When enabled, set filter conditions, partition filters, or standard data filters for the comparison table. These filters append directly to the validation SQL.
Validation Configuration
Rule Validation
After validation, Dataphin compares results against exception validation settings. If conditions match, the result is marked as failed. Dataphin then triggers alerts and other follow-up actions.
Available exception validation metrics depend on the template and configuration. You can combine up to three conditions using AND or OR logic.
For more information, see Validation Configuration.
Archiving Configuration
Exception Archiving
Disabled by default. When enabled, Dataphin archives exception data to a file or table. After quality validation, download and analyze archived exception data.
Archive Mode supports Archive Exception Fields Only and Archive Full Record.
Archive Exception Fields Only: Archives only the current monitored field after deduplication. Use this when a single field fully identifies exception data.
Archive Complete Records: Archives the entire record that contains abnormal data. Use this mode when you must examine the complete record to identify abnormal data.
Note: Because archiving complete records significantly increases the archived data volume, we recommend using Archive Abnormal Fields Only under normal circumstances.
Archive Location supports Default File Server and Archive Table For Exception Data. If no exception archiving table exists, click Manage Exception Archiving Tables to create one. For more information, see Add an Exception Archiving Table.
Default File Server: The system file server configured during Dataphin deployment. You can download exception data directly from the Validation Records > Validation Details page. Or access the default file server directly. With the default file server, Dataphin archives up to 100 exception records per validation run. Best for small-data validation.
Archive Table For Exception Data: Use this to store more exception data or group exceptions from different validation runs for comparative analysis. Each quality rule stores up to 10,000 exception records per run. You can download exception data quickly from the validation records page. Or access the archive table directly and customize its lifecycle for greater flexibility.
NoteDataphin downloads exception data for all rules triggered in the current run. Download limit: 10,000 records. To view more data, archive to a custom exception archiving table and access it directly.
Your exception archiving table must meet specific format requirements. Otherwise, write errors may occur. For details, see Add an Exception Archiving Table.
Business Attribute Configuration
Attribute Information
Business attribute input rules depend on quality rule attribute settings. Example: The field for the responsible department uses an enumeration (multi-select) with values such as Big Data Department, Business Department, and Technology Department. So, when creating a quality rule, this attribute appears as a multi-select dropdown with those options.
The field for the rule owner accepts custom text input. Maximum length: 256 characters. So, when creating a quality rule, enter up to 256 characters.
If the attribute uses a Range input method, configure it as follows:
Range: Used for continuous numeric or date ranges. Select from >, >=, <, or <=. For more attribute configuration options, see Create and Manage Quality Rule Attributes.
Scheduling Configuration
Scheduling Method
Select an existing schedule. If you have not decided on a schedule yet, create the quality rule first and configure scheduling later. To create a new schedule, see Create a Schedule.
Quality Score Configuration
Scoring Method
Supports two scoring methods: quality check status and data pass rate.
Validation Status: Scores based on the most recent successful validation. Pass = 100 points. Fail = 0 points.
Data Pass Rate: Scores based on the percentage of valid data in the most recent successful validation. For example, if 80% of data passes format validation, the quality score is 80.
Not all rule templates support both methods. Templates that support only Validation Status include the following:
Uniqueness: Field Group Count Check and Field Duplicate Value Count Check.
Consistency: Single-Table Field Statistical Value Consistency Check and Cross-Source Two-Table Field Statistical Value Consistency Check.
Stability Rule Categories.
Custom SQL: Custom Statistical Metric Check.
Quality Score Weight
Set the weight for this quality rule’s score. Dataphin uses this to calculate the overall quality score for the monitored object. Enter an integer from 1 to 10.
You can click OK to finish configuring the rule.
Click Preview SQL to compare the current configuration with the last saved version and identify any changes to the SQL.
NotePreview SQL is unavailable when required fields are incomplete.
The left side shows the SQL preview for the last saved configuration. If no previous configuration exists, it is blank. The right side shows the SQL preview for the current configuration.
Rule Configuration List
View, edit, trial run, execute, and delete configured data table rules on the rule configuration list page.
region | Description |
① Filter and Search Area | Search quickly by object name or rule name. Filter by rule type, rule template, rule strength, trial-run status, or active status. Note If a business attribute is configured as searchable and filterable and is enabled, you can search or filter by that attribute. |
② List Area | Displays the object type/name, rule name/ID, test run status, activation status, rule type, rule template, rule strength, schedule type, and related knowledge base document information in the rule configuration list. Click the refresh icon
|
③ Actions Area | Perform view, clone, edit, trial-run, execute, schedule configuration, associate knowledge base document, quality score configuration, and delete actions.
|
④ Batch Actions Area | Perform batch trial-run, execute, schedule configuration, enable, disable, update business attributes, associate knowledge base document, quality score configuration, export rules, and delete actions.
|
to select the fields to display in the rule configuration list.
icon View Trial Run Log.Create a Schedule
When configuring scheduling for a rule, you can use existing schedules (maximum 20 per table).
You can assign up to 10 schedules to a single rule.
Duplicate schedules are automatically removed.
The validation scope becomes a filter condition in the validation SQL, controlling the data range for each validation. It also serves as the basic unit for downstream quality reports, which use the validation scope as the smallest viewing unit.
On the Quality Rule Details page, click the Scan Configuration tab. Then, click Create Schedule to open the Create Schedule dialog box.
In the Create Schedule dialog box, you can configure the parameters.
Parameter
Description
Schedule Name
Enter a custom schedule name. Maximum length: 64 characters.
Schedule Type
Supports two types of scheduling: Timed Scheduling and Fixed-task Trigger Scheduling.
Recurrency Triggered: Runs quality checks on a fixed schedule. Best for data with predictable output times.
Recurrence: Quality rule execution consumes computing resources. Avoid concurrent execution of many rules at once. This could affect production tasks. Recurrence options: Day, Week, Month, Hour, and Minute.
If the system time zone (user center time zone) differs from the schedule time zone (set in Management Center > System Settings > Basic Settings), the rule runs in the system time zone.
Task Triggered: Runs the quality rule after or before a specified task completes. Supported task types: Engine SQL, Offline Pipeline, Python, Shell, Virtual, DataX, Spark JAR, Hive MR, and Database SQL Node. Best for tables with fixed modification tasks.
NoteTask-triggered schedules only support production environment tasks. If you set a strong rule and the schedule fails, it may affect online tasks. Proceed with caution based on business needs.
Trigger Timing: Choose when to run the quality check. Options: Trigger after all tasks succeed, Trigger after each task succeeds, and Trigger before each task runs.
Triggering Task: Project administrators or O&M system roles can select task nodes from production projects. Search by output name or choose from recommended or all tasks.
NoteIf you select Trigger after all tasks succeed, choose triggering tasks with the same schedule period. This avoids delayed rule execution and late quality report generation.
Schedule Condition
Disabled by default. When enabled, Dataphin checks whether conditions are met before scheduling. Scheduling proceeds only if conditions are met. Otherwise, Dataphin skips the schedule.
Business Date/Executed On: Available for Recurrency Triggered (not supported for timed scheduling) and Task Triggered. Select Standard Calendar or Custom Calendar. To create a custom calendar, see Create a Public Calendar.
If you select Standard Calendar, choose from Month, Week, and Date. See the following image:
If you select Custom Calendar, choose from Date Type and Tag. See the following image:
Instance Type: Available for Task Triggered. Select Recurring Instance, Data Backfill Instance, or One-Time Instance. See the following image:
NoteConfigure at least one condition. To add a condition, click + Add Rule.
Maximum: 10 schedule conditions.
Set relationships between conditions as AND or OR.
Validation Scope Expression
Enter or select a validation scope expression. For example:
ds='${yyyyMMdd}'. You can also select a built-in expression and modify it. For built-in expressions, see Built-in Partition Filter Expressions.NoteUse and or or to combine multiple conditions. Example:
province="Zhejiang" and ds<=${yyyyMMdd}.If you configure filter conditions in the quality rule, Dataphin applies them with AND logic to the validation scope expression. Both conditions filter the data.
The validation scope expression supports full-table scans.
Note: Full-table scans consume significant resources. Some systems do not support them. Use partition filter expressions instead.
Validation Scope Budget
Defaults to the current business date.
Click OK to finish configuring the schedule.
Schedule Configuration List
After creating a schedule, view, edit, clone, or delete it in the schedule configuration list.
Region | Description |
① Filter and Search Area | Search quickly by schedule name. Filter by Recurrency Triggered or Task Triggered. |
② List Area | Shows Schedule Name, Schedule Type, Last Updated By, and Last Updated At. |
③ Actions Area | Edit, clone, or delete schedules.
|
Configure Alerting
Assign different alerting methods to different rules. For example, you can use phone alerts for strong rule failures and SMS alerts for weak rule failures. If a rule matches multiple alert configurations, set an alert priority policy.
You can create up to 20 alert configurations per monitored object.
On the Quality Rule Details page, click the Alert Configuration tab. Then click Create Alert Configuration. The Create Alert Configuration dialog box opens.
In the Create Alert Configuration dialog box, configure the parameters.
Parameter
Description
Scope
Select All Rules, All Strong Rules, All Weak Rules, or Custom.
NotePer monitored object, you can configure one alert for All Rules, one for All Strong Rules, and one for All Weak Rules. New rules auto-match based on strength. To change an alert, edit the existing configuration.
Custom scope supports up to 200 rules from the current monitored object.
Alert Configuration Name
Unique per monitored object. Maximum length: 256 characters.
Alert Recipients
Configure recipients and alert methods. Select at least one recipient and one method.
Alert Recipient: You can select from three types of alert recipients: custom, on-call schedule, and quality owner.
Supports up to 5 custom recipients and up to 3 on-call schedules.
Alert Methods: Choose from phone, email, text message, DingTalk, Lark, WeCom, or Custom Channel. Manage these methods in Channel Settings.
Click OK to complete the alert configuration.
Alert Configuration List
After configuration, sort, edit, or delete alerts in the alert configuration list.
Number | Description |
① Sort Area | Set alert priority when a rule matches multiple alert configurations:
|
② List Area | Shows alert configuration name, scope, recipients per alert type, and corresponding alert methods. Scope: For custom alerts, view object and rule names. If the rule is deleted, the object name is unavailable. Update the alert configuration as needed. |
③ Actions Area | Edit or delete alerts.
|
icon next to alert names, or use the icons in the Actions column. Icons from left to right: Pin to Top, Pin to Bottom. After sorting, click Sort Complete.
Add an Exception Archiving Table
An exception archiving table stores records of quality rule validation failures.
On the Quality Rule Details page, click the Archive tab. Then click + Add Exception Archiving Table. The Add Exception Archiving Table dialog box opens.
In the Add Exception Archiving Table dialog box, configure the parameters.
Method supports Create Table and Select Existing Table. These methods add special quality validation fields. Archived exception data is never written to original data tables.
Create Table: Enter a custom table name. The default name is current_table_name_exception_data. The table is created in the same project or section as the archiving table. Valid characters include letters, numbers, underscores (_), and periods (.). The maximum length is 128 characters.
If the monitored table is a physical table, the archiving table is created in the same project.
If the monitored table is a logical dimension table or logical fact table, the archiving table defaults to the same project. You can manually specify a project in the same section, such as `projectA.table_name`.
If the monitored table is a logical aggregate table, specify a project in the same section for the archiving table name. Otherwise, Dataphin creates it in a project within the monitored table’s section.
The archiving table must include all fields from the quality monitoring table, plus validation fields. The script format is:
create table current_table_name_exception_data (dataphin_quality_tenant_id varchar(64) comment 'Tenant ID' , dataphin_quality_rule_id varchar(64) comment 'Quality Rule ID', dataphin_quality_rule_name varchar(256) comment 'Quality Rule Name', dataphin_quality_column_name varchar(1024) comment 'Validation Field Name', dataphin_quality_watch_task_id varchar(128) comment 'Monitored Object Task ID', dataphin_quality_rule_task_id varchar(64) comment 'Rule Task ID', dataphin_quality_validate_time varchar(64) comment 'Quality Validation Time', dataphin_quality_archive_mode varchar(32) comment 'Exception Archive Mode, ONLY_ERROR_FIELD/FULL_RECORD', dataphin_quality_error_data string comment 'Exception Data', ljba_id bigint comment 'ljba Primary Key', ljb_id bigint comment 'ljb Primary Key', col_tinyint tinyint comment 'Field Type TINYINT (lowercase)', col_tinyint_02 tinyint comment '2', col_smallint smallint comment 'Field Type SMALLINT (lowercase)', col_smallint_02 smallint comment '4', col_int int comment 'Field Type INT (lowercase)', col_int_02 int comment '6', col_bigint bigint comment 'Field Type BIGINT (lowercase)', col_bigint_02 bigint comment '8', col_float float comment 'Field Type FLOAT (lowercase)', col_float_02 float comment '10', col_double double comment 'Field Type DOUBLE (lowercase)', col_double_02 double comment '11', col_decimal decimal(38,18) comment 'Field Type DECIMAL(38,18) (lowercase)', col_decimal_02 decimal(38,18) comment '12', col_varchar varchar(500) comment 'Field Type VARCHAR(500) (lowercase)', col_varchar_02 varchar(500) comment '13', col_char char(10) comment 'Field Type CHAR(10) (lowercase)', col_char_02 char(10) comment '14', col_string string comment 'Field Type STRING (lowercase)', col_string_02 string comment '15', col_date date comment 'Field Type DATE (lowercase)', col_date_02 date comment '16', col_datetime datetime comment 'Field Type DATETIME (lowercase)', col_datetime_02 datetime comment '17', col_timestmap timestamp comment 'Field Type TIMESTAMP (lowercase)', col_timestmap_02 timestamp comment '18', col_boolean boolean comment 'Field Type BOOLEAN (lowercase)', col_boolean_02 boolean comment '19', col_binary binary comment 'Field Type BINARY (lowercase)', col_binary_02 binary comment '20', col_array array<int> comment 'Field Type ARRAY<int> (lowercase)', col_array_02 array<string> comment '21', col_map map<string,string> comment 'Field Type MAP<string, string> (lowercase)', col_map_02 map<string,int> comment '22', ds string comment 'Date Partition, yyyyMMdd' ) partitioned by (dataphin_quality_validate_date string comment 'Validation Date (Partition Field)');
Select Existing Table: Select a table from the same project or data source. The archiving table must contain all fields from the quality monitoring table and the validation fields. Click View DDL for Exception Archiving Table to view the CREATE TABLE statement. Script format:
create table current_table_name_exception_data (dataphin_quality_tenant_id varchar(64) comment 'Tenant ID' , dataphin_quality_rule_id varchar(64) comment 'Quality Rule ID', dataphin_quality_rule_name varchar(256) comment 'Quality Rule Name', dataphin_quality_column_name varchar(1024) comment 'Validation Field Name', dataphin_quality_watch_task_id varchar(128) comment 'Monitored Object Task ID', dataphin_quality_rule_task_id varchar(64) comment 'Rule Task ID', dataphin_quality_validate_time varchar(64) comment 'Quality Validation Time', dataphin_quality_archive_mode varchar(32) comment 'Exception Archive Mode, ONLY_ERROR_FIELD/FULL_RECORD', dataphin_quality_error_data string comment 'Exception Data', ljba_id bigint comment 'ljba Primary Key', ljb_id bigint comment 'ljb Primary Key', col_tinyint tinyint comment 'Field Type TINYINT (lowercase)', col_tinyint_02 tinyint comment '2', col_smallint smallint comment 'Field Type SMALLINT (lowercase)', col_smallint_02 smallint comment '4', col_int int comment 'Field Type INT (lowercase)', col_int_02 int comment '6', col_bigint bigint comment 'Field Type BIGINT (lowercase)', col_bigint_02 bigint comment '8', col_float float comment 'Field Type FLOAT (lowercase)', col_float_02 float comment '10', col_double double comment 'Field Type DOUBLE (lowercase)', col_double_02 double comment '11', col_decimal decimal(38,18) comment 'Field Type DECIMAL(38,18) (lowercase)', col_decimal_02 decimal(38,18) comment '12', col_varchar varchar(500) comment 'Field Type VARCHAR(500) (lowercase)', col_varchar_02 varchar(500) comment '13', col_char char(10) comment 'Field Type CHAR(10) (lowercase)', col_char_02 char(10) comment '14', col_string string comment 'Field Type STRING (lowercase)', col_string_02 string comment '15', col_date date comment 'Field Type DATE (lowercase)', col_date_02 date comment '16', col_datetime datetime comment 'Field Type DATETIME (lowercase)', col_datetime_02 datetime comment '17', col_timestmap timestamp comment 'Field Type TIMESTAMP (lowercase)', col_timestmap_02 timestamp comment '18', col_boolean boolean comment 'Field Type BOOLEAN (lowercase)', col_boolean_02 boolean comment '19', col_binary binary comment 'Field Type BINARY (lowercase)', col_binary_02 binary comment '20', col_array array<int> comment 'Field Type ARRAY<int> (lowercase)', col_array_02 array<string> comment '21', col_map map<string,string> comment 'Field Type MAP<string, string> (lowercase)', col_map_02 map<string,int> comment '22', ds string comment 'Date Partition, yyyyMMdd' ) partitioned by (dataphin_quality_validate_date string comment 'Validation Date (Partition Field)');
Click OK to add the exception archiving table.
Select Automatically Set as Active Archiving Table After Creation to simplify the creation of quality rules.
View Exception Archiving Table List
After adding a table, the first one is active by default. Click its name to view schema details. You can also set other tables as active or delete them.
Set as Active Archiving Table: Sets the table as active. You can use this table to archive exception data for all quality rules associated with this monitored object that specify a custom exception archiving table.
Delete: Removes only the reference to the exception archiving table. The table itself remains intact. You can re-add it later if needed.
View Quality Reports
Click Quality Report to view Rule Validation Overview and Rule Validation Details.
Filter validation details quickly by exception result, partition time, or keywords in rule or object names.
In the validation details list, click the
icon in the Actions column to view rule validation details.In the validation details list, click the
icon in the Actions column to view execution logs.
Configure Quality Rule Permission Management
Click Permission Management. Configure View Details to specify who can view validation records, quality rule details, and quality reports.
View Details: Choose All Members or Members with Quality Management Permissions for This Object.
You can click OK to complete the permission management configuration.
What to Do Next
After completing the quality rule configuration, you can view it on the enterprise-wide data table rule list page. For more information, see View Monitored Object List or .