Dataphin:Create metric quality rules

Prerequisites

You must add monitored objects before you can configure quality rules. For more information about how to add monitored objects, see Add monitored objects.

Permissions

• Super administrators, quality administrators, custom global roles with Quality Rule - Manage permissions, custom project roles with Project Quality Management - Quality Rule Management permissions for the project that contains the table, and metric business owners can configure scheduling and alerts for quality rules.

• Quality owners and regular users require additional read permissions for logical table fields. To request these permissions, see Apply for, renew, and return table permissions.

• The supported operations vary depending on the object. For more information, see Quality rule operation permissions.

Validation rule details

When metrics are validated against quality rules, the system sends an alert message if a soft rule is triggered. This helps you promptly find and handle exceptions. If a strong rule is triggered, the system automatically interrupts the task that contains the table to prevent dirty data from flowing to downstream nodes. The system also sends an alert message to help you find and handle the exception.

Trial runs vs. executions

The difference between a trial run and an execution lies in the execution method and how the results are handled. A trial run is a test execution of a quality rule to check its correctness and performance. The results of a trial run are not displayed in the quality report. An execution runs a quality rule check at a specified time, and the results are sent to the quality report for you to view and analyze.

Configure a quality rule

1. On the Dataphin homepage, choose Administration > Data Quality from the top menu bar.

2. In the navigation pane on the left, click Quality Rule. On the Metrics page, click the name of the target object to go to the Quality Rule Details page, where you can configure the quality rule.

3. On the Quality Rule Details page, click Create Quality Rule.

4. In the Create Quality Rule dialog box, configure the parameters.

   Parameter		Description
   Basic information	Rule Name	Enter a custom name for the quality rule.
   	Rule Strength	You can select Weak Rule or Strong Rule. If you select Weak Rule, an alert is triggered if the validation result is abnormal. However, downstream nodes are not blocked. If you select Strong Rule, an alert is triggered if the validation result is abnormal. If downstream tasks exist, such as code check scheduling or task-triggered scheduling, the downstream tasks are blocked to prevent data contamination. If no downstream tasks exist, such as for periodic quality scheduling, only an alert is triggered.
   	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
   	Configuration Method	Create From Template: Quickly create quality rules using general system templates and custom business templates. System Template: The built-in parameters are configurable. This is suitable for creating general rules. Custom Template: The parameters are preset and do not require configuration. This is typically used for creating rules that contain business logic. SQL: Flexibly customize quality monitoring rules with SQL. This is suitable for flexible and complex scenarios.
   	Rule Template	Select a rule template from the drop-down list. Options are Uniqueness, Stability, and SQL. Uniqueness: Includes Field Group Count Validation and Duplicate Value Count Validation. Stability: Includes Column Stability Validation and Column Volatility Validation. SQL: Includes Custom Statistic Validation. For more information, see Quality rule template types.
   	Rule Type	The rule type depends on the template and is its most basic property. It can be used for descriptions and filtering.
   Rule configuration	Rule Configuration	If you set Rule Template to Uniqueness, configure the following parameters. Field Group Count Validation/Duplicate Value Count Validation: Validation Table Data Filtering: This feature is disabled by default. If you enable it, you can configure filter conditions, partition filters, or regular data filters for the validation table. The filter conditions are appended to the validation SQL statement. If the validation table requires partition filtering, configure a partition filter expression in the schedule configuration. After configuration, the quality report uses the validation partition as the minimum granularity for viewing. Enter the data filtering content. Examples: id = 12 --single table T1.id=12 and T2.name = "Zhang San" --double table If you set Rule Template to Stability, configure the following parameters. Column Stability Validation/Column Volatility Validation: Statistical Method: Select a statistical method as needed. Validation Table Data Filtering: This feature is disabled by default. If you enable it, you can configure filter conditions, partition filters, or regular data filters for the validation table. The filter conditions are appended to the validation SQL statement. If the validation table requires partition filtering, configure a partition filter expression in the schedule configuration. After configuration, the quality report uses the validation partition as the minimum granularity for viewing. Enter the data filtering content. Examples: id = 12 --single table T1.id=12 and T2.name = "Zhang San" --double table If you set Rule Template to SQL, configure the following parameters. Custom Statistic Validation: SQL: SELECT query statements are supported. The query object must include the primary table. Example: select sum(sale) from tableA where ds=${bizdate};
   Validation configuration	Rule Validation	After a data quality rule is validated, the result is compared with the exception check configuration. If the conditions are met, the validation fails. This also triggers subsequent processes such as alerts. The available metrics for exception checks are determined by the template and configuration. Multiple AND/OR conditions are supported. We recommend using fewer than three conditions in your configuration. For more information, see Metric validation configuration.
   Business property configuration	Property Information	The standards for entering business properties depend on the quality rule property configuration. Examples: The value type for the field corresponding to the managing department is an enumeration (multiple selection). The selectable values are Big Data Department, Business Department, and Technical Department. Therefore, when you create a quality rule, this property is a multi-select drop-down list with these options. The value type for the field corresponding to the rule owner is custom input. The property field length is 256. Therefore, when you create a quality rule, you can enter up to 256 characters for this property. If the property field is filled using a range selection, configure it as follows: Range selection: This is often used when the value range consists of continuous numbers or dates. You can use the symbols >, >=, <, and <=. For more information about property configuration, see Create and manage quality rule properties.
   Scheduling property configuration	Scheduling Method	Select a configured schedule. If you have not decided on a scheduling method, you can configure it after creating the quality rule. To create a new schedule, see Create a schedule.

5. Click Save to complete the rule configuration.

   You can click Preview SQL to compare the current configuration with the last saved configuration. This lets you view the changes in the SQL statement.

   Note

   • The SQL preview is unavailable if key information is incomplete.

   • The left side shows the SQL preview of the last saved configuration. This section is empty if no configuration has been saved. The right side shows the SQL preview of the current configuration.

   Rule configuration list

   On the rule configuration list, you can view information about the configured metric rules and perform operations such as view, edit, trial run, run, and delete.

   

   Area	Description
   ①Filter and search area	Lets you quickly search by object or rule name. Lets you filter by rule type, rule template, rule strength, trial run status, and active status. Note If a quality rule property is configured with searchable and filterable business properties and is enabled, you can search or filter based on this property.
   ②List area	Displays the object type/name, rule name/ID, trial run status, active status, rule type, rule template, rule strength, schedule type, and related knowledge base document information from the rule configuration list. Click the icon before the refresh icon to select the fields to display in the rule list. Active Status: We recommend running a trial run before activating a rule. Activate the status for rules that pass the trial run to prevent incorrect rules from blocking online tasks. After you activate the status, the selected rules automatically run based on the configured schedule. After you deactivate the status, the selected rules do not run automatically but can be run manually. Related Knowledge Base Document: Click View Details to view the knowledge base information associated with the rule. This includes the table name, validation object, rule, and related knowledge base document information. You can also search, view, edit, and delete the knowledge base. For more information, see View a knowledge base.
   ③Operation area	You can perform operations such as view, clone, edit, trial run, run, schedule configuration, associate knowledge base document, and delete. View: View the details of the rule configuration. Clone: Quickly clone a rule. Edit: After you edit a rule, you must run a trial run again. Trial Run: Select Existing Schedule or Custom Validation Range to run a trial run of the rule. After the trial run, click the icon to View The Trial Run Log. Run: Select Existing Schedule or Custom Validation Range to run the rule. After the run, you can view the validation results in Quality Record. Schedule Configuration: In the dialog box, filter schedule types or quickly search for schedules by name. You can also edit schedules. Associate Knowledge Base Document: After you associate a rule with a knowledge base, you can view the associated knowledge in the quality rule and administration workbench. You can select unassociated knowledge bases. To create one, see Create and manage a knowledge base. Delete: Deleting this quality rule object deletes all quality rules under it. This action cannot be undone. Proceed with caution.
   ④Batch operation area	You can perform batch operations such as trial run, run, configure schedule, enable, disable, modify business properties, associate knowledge base document, and delete. Trial Run: Select Existing Schedule or Custom Validation Range to run a batch trial run of rules. After the trial run, click the icon to View The Trial Run Log. Run: Select Existing Schedule or Custom Validation Range to run a batch of rules. After the run, you can view the validation results in Quality Record. Schedule Configuration: In the dialog box, filter schedule types or quickly search for schedules by name. You can also edit schedules to configure them for quality rules in batches. You can only modify selected rules that are editable on the quality rule list page. Enable: After you enable the active status in batches, the selected rules automatically run based on the configured schedule. You can only enable selected rules that are editable on the quality rule list page. Disable: After you disable the active status in batches, the selected rules do not run automatically but can be run manually. You can only disable selected rules that are editable on the quality rule list page. Modify Business Properties: When the value type for the business property is single-selection or multiple-selection, you can modify business properties in batches. If the value type is multiple-selection, you can append or modify property values. If the value type is single-selection, you can directly modify property values. Associate Knowledge Base Document: After you associate rules with a knowledge base, you can view the associated knowledge in the quality rule and administration workbench. You can configure knowledge bases for monitored objects in batches. To create one, see Create and manage a knowledge base. Delete: Lets you delete quality rule objects in batches. This action cannot be undone. Proceed with caution. You can only delete selected rules that are editable on the quality rule list page.

Create a schedule

1. On the Quality Rule Details page, click the Schedule Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

2. In the Create Schedule dialog box, configure the parameters.

   Parameter	Description
   Schedule Name	Enter a custom name for the schedule.
   Schedule Type	You can select Timed Scheduling, Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling. Timed Scheduling: Runs periodic quality checks on data based on the set schedule. This is suitable for scenarios where data is generated at a relatively fixed time. Scheduling Cycle: Running quality rules consumes computing resources. Avoid running multiple quality rules at the same time to ensure the normal operation of production tasks. The scheduling cycle can be Day, Week, Month, Hour, or Minute. If the system time zone (the time zone in the User Center) is different from the scheduling time zone (the time zone configured in Management Hub > System Settings > Basic Settings), the rule runs based on the system time zone. Data Update-triggered Scheduling: When any code task runs, the system checks if the run updates the specified validation scope of the current table. This is suitable for tables with non-fixed modification tasks or tables that require close monitoring, where every change must be monitored. Note We recommend selecting the partition updated by the task as the validation scope. For non-partitioned tables, the entire table is checked. The system automatically detects all data changes and performs checks to avoid omissions. Fixed Task-triggered Scheduling: Runs the configured quality rules before or after a specified task runs successfully. You can select tasks of types such as SQL, Offline Pipeline, Python, Shell, Virtual, DataX, Spark Jar, Hive MR, and Database SQL to trigger the quality rule task. This is suitable for scenarios where table modification tasks are fixed. Note Fixed task triggers can only select production environment tasks. If a strong rule is configured and the scheduled task fails the check, online tasks may be affected. Proceed with caution based on your business needs. Trigger Timing: Select when to trigger the quality check. You can select Trigger After All Tasks Run Successfully, Trigger After Each Task Runs Successfully, or Trigger Before Each Task Runs. Triggering Task: Select a task node from the production project. This option is available to project administrators and users with the O&M system role. You can search by node output name, or select from recommended or all tasks. Note If you select to trigger after all tasks run successfully, we recommend selecting tasks with the same scheduling cycle. This avoids delays in rule execution and quality check results due to different scheduling cycles.
   Scheduling Condition	This feature is disabled by default. If you enable it, the system checks if the scheduling conditions are met before the quality rule is scheduled. The rule is scheduled only if the conditions are met. Otherwise, the schedule is ignored. Data Timestamp/Executed On: If you select Timed Scheduling (which does not support execution date), Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling as the schedule type, you can configure the date. You can select Regular Calendar or Custom Calendar. For more information about how to customize a calendar, see Create a public calendar. If you select Regular Calendar, you can set conditions for Month, Week, and Date. See the following figure. If you select Custom Calendar, you can set conditions for Date Type and Tag. See the following figure. Instance Type: If you select Data Update-triggered Scheduling or Fixed Task-triggered Scheduling as the schedule type, you can configure the instance type. You can select Recurring Instance, Data Backfill Instance, or One-time Instance. See the following figure. Note Configure at least one rule. To add a rule, click +Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.
   Validation Scope	If you select timed scheduling or fixed task-triggered scheduling, the validation scope supports custom validation scope. If you select data update-triggered scheduling, the validation scope supports partitions updated by the task and custom validation scope. Updated Partition: If a partition is updated in the check task, the task is issued based on the updated partition. Note In dynamic partition scenarios, the partition may not be parsed, and no quality check is performed. Volatility check rules, such as checking partition size, partition row count, and field statistics, require a specified partition and do not support the task-updated partition validation scope. If a non-partitioned table is updated, the entire table is checked. Custom validation scope: For scenarios that cannot be parsed, you can use a custom validation scope to specify the validation scope expression based on the data timestamp or execution date. Validation Scope Expression: This is a drop-down list where you can enter the scope to check, such as ds='${yyyyMMdd}'. You can also select a built-in validation scope expression and modify it for quick configuration. For more information about partition expressions, see Built-in partition expression types. Note If multiple conditions are used for the check, you can connect them with and or or, such as province="Zhejiang" and ds<=${yyyyMMdd}. If a filter condition is configured in the quality rule, the validation scope expression and the filter condition have an AND relationship. When data is checked, both conditions are used for filtering. The validation scope expression supports full table scans. Note: A full table scan consumes significant resources, and some tables do not support it. We recommend configuring a partition filter expression to avoid a full table scan. Validation Scope Budget: The default is the data timestamp of the current day.

3. Click OK to complete the schedule configuration.

Scheduling configuration list

After you create a schedule, you can view, edit, clone, or delete it in the scheduling configuration list.

Configure an alert

You can configure different alert methods for different rules to differentiate alerts. For example, you can configure phone alerts for exceptions from strong rules and text message alerts for exceptions from soft rules. If a rule matches multiple alert configurations, you can set a policy to determine which alert takes effect.

1. On the Quality Rule Details page, click the Alert Configuration tab, and then click New Alert Configuration to open the New Alert Configuration dialog box.

   

2. In the New Alert Configuration dialog box, configure the parameters.

   Parameter	Description
   Coverage	You can select All Rules, All Strong Rules, All Weak Rules, or Custom. Note For a single monitored object, you can configure one alert for each of the three ranges: all rules, all strong rules, and all soft rules. New rules automatically match the corresponding alert based on their strength. To change an alert configuration, you can modify the existing one. The custom range can include all configured rules under the current monitored object, up to a maximum of 200.
   Alert Configuration Name	The alert configuration name must be unique for a single monitored object and can be up to 256 characters long.
   Alert Recipient	Configure the alert recipients and alert method. You must select at least one alert recipient and one alert method. Alert Recipient: You can select custom, shift schedule, or quality owner as the alert recipient. You can configure up to 5 custom alert recipients and up to 3 shift schedules. Alert Method: You can select different receiving methods, such as phone, email, text message, DingTalk, Lark, WeCom, and custom channel. This receiving method can be controlled through Configure a Custom Channel.

3. Click OK to complete the alert configuration.

Alert configuration list

After you complete the alert configuration, you can sort, edit, or delete it in the alert configuration list.

View a quality report

Click Quality Report to view the Rule Check Overview and Rule Check Details of the current quality rule.

• You can quickly filter check details by exception result, partition time, or keywords in rule or object names.

• In the Actions column of the rule check details list, click the icon to view the rule check details of the quality rule.

• In the Actions column of the rule check details list, click the icon to view the execution log of the quality rule.

Set quality rule permissions

1. Click Permission Management and configure the View Details permission to specify which members can view check records, quality rule details, and quality reports.

   View Details: You can select All Members or Only Members With Quality Management Permissions For The Current Object.

2. Click OK to complete the permission management configuration.

What to do next

After you configure the quality rule, you can view it on the metric rule list. For more information, see View the monitored object list.