Dataphin:Create A Metric Quality Rule

Prerequisites

You must add a monitored object before you can configure a quality rule. For more information about how to add a monitored object, see Add a monitored object.

Permissions

• The following users can configure schedules, alerts, and runtime configurations for quality rules: super administrators, quality administrators, users with a custom global role that has Quality Rules-Manage permissions, users with a custom project role that has Project Quality Management-Quality Rule Management permissions for the table's project, and metric business owners.

• Quality owners and regular users must also have read permissions on logical table fields. To apply for these permissions, see Apply for, Renew, and Return Table Permissions.

• Supported operations vary by object. For more information, see Quality Rule Operation Permissions.

How rule checks work

When a metric is checked against a quality rule, if a weak rule is triggered, the system sends you an alert to help you promptly identify and address the anomaly. If a strong rule is triggered, the system automatically blocks the associated task to prevent dirty data from flowing downstream and sends you an alert.

Trial run vs. run

The difference between a trial run and a run lies in how they are executed and where their results are displayed. A trial run is a test execution of a quality rule to verify its correctness and behavior. The results of a trial run do not appear in the quality report. A run checks a quality rule at a specific time, and its results are included in the quality report.

Quality rule configuration

1. On the Dataphin homepage, choose Governance > Data Quality from the top navigation bar.

2. In the left navigation pane, click Quality Rules. On the Metric page, click the name of the target object to open the Quality Rule Details page, where you can configure the quality rule.

3. On the Quality Rule Details page, click Create Quality Rule.

4. In the Create Quality Rule dialog box, configure the parameters.

   Parameter		Description
   Basic information	Rule name	Enter a custom name for the quality rule.
   	Rule severity	You can select Weak Rule or Strong Rule. If you select Weak Rule, an alert is triggered when the rule check fails, but downstream tasks are not blocked. If you select Strong Rule, an alert is triggered when the rule check fails. If there are downstream tasks (such as those triggered by a code check schedule or a task trigger schedule), they are blocked to prevent dirty data from flowing downstream. If there are no downstream tasks (as in a periodic quality schedule), only an alert is triggered.
   	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
   	Configuration method	Create from Template: Creates a quality rule by using a general-purpose system template or a custom business template. System Template: Suitable for creating general-purpose rules. You can configure the template's built-in parameters. Custom Template: Suitable for creating rules that involve business logic. The template's parameters are preset and cannot be configured. Custom SQL: Defines quality monitoring rules by using SQL queries. This method is ideal for complex scenarios.
   	Rule template	Select a rule template from the drop-down list: Uniqueness, Stability, or Custom SQL. Uniqueness: Includes Field Group Count Check and Duplicate Field Value Count Check. Stability: Includes Field Stability Check and Field Fluctuation Check. Custom SQL: Includes Custom Statistical Metric Check. For more information, see Quality rule template types.
   	Rule type	The rule type depends on the template. It is a fundamental attribute of a template, used for description and filtering.
   Rule configuration	Rule configuration	If you set Rule Template to Uniqueness, the following parameters are available. Field Group Count Check/Duplicate Field Value Count Check: Filter for Checked Table: Disabled by default. When enabled, you can configure filter conditions for the table being checked, such as partition filters or general data filters. The filter conditions are appended directly to the check SQL. If the table requires partition filtering, we recommend that you configure a partition expression in the scheduling configuration. After configuration, you can view the quality report at the partition level. Enter the data filter content, for example: id = 12 -- For a single table T1.id=12 and T2.name = "Zhang San" -- For two tables If you set Rule Template to Stability, the following parameters are available. Field Stability Check/Field Fluctuation Check: Statistic Method: We recommend that you select a statistic method based on your business scenario. Filter for Checked Table: Disabled by default. When enabled, you can configure filter conditions for the table being checked, such as partition filters or general data filters. The filter conditions are appended directly to the check SQL. If the table requires partition filtering, we recommend that you configure a partition expression in the scheduling configuration. After configuration, you can view the quality report at the partition level. Enter the data filter content, for example: id = 12 -- For a single table T1.id=12 and T2.name = "Zhang San" -- For two tables If you set Rule Template to Custom SQL, the following parameters are available. Custom Statistical Metric Check: Custom SQL: Only SELECT statements are supported, and the query object must include the main table. Example: select sum(sale) from tableA where ds=${bizdate};
   Check configuration	Rule check	After a data quality rule check completes, the system compares the result with the exception criteria you configured. If the criteria are met, the check fails and triggers subsequent processes such as alerts. Available exception metrics depend on the template and your configuration. You can use multiple conditions with AND/OR logic. We recommend using fewer than three conditions. For more information, see Metric check configuration.
   Business attribute configuration	Attribute information	The format for business attributes depends on the quality rule attribute configuration. Examples: The field corresponding to the managing department is of an enumerated type (multiple selection), with possible values being Big Data Department, Business Department, and Technology Department. Therefore, when creating a quality rule, this attribute is a multi-select drop-down list with these options. The rule owner field is a custom input with a maximum length of 256 characters. Therefore, when creating a quality rule, you can enter up to 256 characters for this attribute. If the input method for an attribute field is Range, configure it as follows: Range: Used for value ranges of continuous numbers or dates. You can use the symbols >, >=, <, and <=. For more information about attribute configurations, see Create and manage quality rule attributes.
   Scheduling attribute configuration	Scheduling method	You can select an existing schedule. If you have not yet selected a scheduling method, you can create the quality rule first and configure the schedule later. To create a new schedule, see Create a schedule.

5. Click Save to complete the rule configuration.

   You can click Preview SQL to compare the SQL code of your current configuration with the last saved version and view the changes.

   Note

   • The SQL Preview is unavailable because the required information is incomplete.

   • The left pane shows the SQL preview of the last saved configuration. If a configuration has not been saved, this pane is empty. The right pane shows the SQL preview of the current configuration.

   Rule configuration list

   On the rule configuration list page, you can review configured metric rules and perform operations on them, such as view, edit, trial run, run, and delete.

   

   Area	Description
   ① Filter and search area	You can perform a quick search by object or rule name. You can filter by rule type, rule template, rule severity, trial run status, or enabled status. Note If quality rule attributes include business attributes that are searchable and filterable and are enabled, you can also search or filter by those attributes.
   ② List area	This area displays the object type/name, rule name/ID, trial run status, enabled status, rule type, rule template, rule severity, schedule type, related knowledge base documents, and last updater/update time for each rule configuration. Click the icon to the left of the refresh button to select the columns to display in the rule list. Enabled status: We recommend that you perform a trial run before enabling a rule. Enable the rule only after the trial run is successful. This helps prevent incorrect rules from blocking production tasks. After you enable a rule, it runs automatically based on the configured schedule. After you disable a rule, it no longer runs automatically but you can run it manually. Related knowledge base document: Click View Details to view the knowledge base information associated with the rule. This includes the table name, check object, rule, and related knowledge base document information. You can also search for, view, edit, and delete knowledge base documents. For more information about these operations, see View a knowledge base.
   ③ Operations area	You can perform the following operations: View, Clone, Edit, Trial Run, Run, Scheduling Configuration, Associate Knowledge Base Document, and Delete. View: View the details of the rule configuration. Clone: Quickly clone a rule. Edit: After editing a rule, you must perform another trial run. Trial Run: You can run a trial run for a rule by using an Existing Schedule or a Custom Check Scope. After the trial run, you can click the icon to View Trial Run Log. Run: You can run a rule by using an Existing Schedule or a Custom Check Scope. After the run, you can view the check results in Check Records. Scheduling Configuration: In the dialog box that appears, you can filter schedules by type or use the schedule name for a quick search. You can also edit schedules. Associate knowledge base document: After you associate a rule with a knowledge base document, you can view the associated knowledge in the quality rule and governance workbench. You can select a knowledge base document that is not already associated. To create one, see Create and manage a knowledge base. Delete: Deleting a quality rule object deletes all quality rules under the object. This action cannot be undone. Proceed with caution.
   ④ Batch operations area	You can perform the following operations in batches: Trial Run, Run, Configure Schedule, Enable, Disable, Modify Business Attributes, Associate Knowledge Base Document, and Delete. Trial Run: You can run a batch trial run for rules by using an Existing Schedule or a Custom Check Scope. After the trial run, you can click the icon to View Trial Run Log. Run: You can run a batch run for rules by using an Existing Schedule or a Custom Check Scope. After the run, you can view the check results in Check Records. Scheduling Configuration: In the dialog box, you can filter schedules by type or use the schedule name for a quick search. You can also edit schedules and configure them for quality rules in batches. You can modify only the selected rules that are editable in the quality rule list. Enable: After you enable rules in a batch, the selected rules run automatically based on their configured schedules. You can enable only the selected rules that are editable in the quality rule list. Disable: After you disable rules in a batch, the selected rules do not run automatically but can be run manually. You can disable only the selected rules that are editable in the quality rule list. Modify business attributes: If the value type for a business attribute is single-select or multi-select, you can batch modify the business attribute. If the value type is multi-select, you can append or replace attribute values. If the value type is single-select, you can directly replace the attribute value. Associate knowledge base document: After you associate rules with a knowledge base document, you can view the associated knowledge in the quality rule and governance workbench. You can batch-configure a knowledge base for monitored objects. To create a knowledge base, see Create and manage a knowledge base. Delete: You can batch-delete quality rule objects. This action cannot be undone. Proceed with caution. You can delete only the selected rules that are editable in the quality rule list.

Create a schedule

1. On the Quality Rule Details page, click the Scheduling Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

2. In the Create Schedule dialog box, configure the parameters.

   Parameter	Description
   Schedule name	Enter a custom name for the schedule.
   Schedule type	You can select a Time-based Schedule, Data Update Trigger, or Fixed Task Trigger. Time-based Schedule: Performs quality checks on data at scheduled, periodic intervals. This is suitable for scenarios where data is generated at relatively fixed times. Schedule period: Quality rule runs consume computing resources. We recommend that you avoid running multiple quality rules concurrently to avoid disrupting production tasks. The schedule period can be set to Day, Week, Month, Hour, or Minute. If the system time zone (the time zone in your user center) is different from the schedule time zone (the time zone configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Data Update Trigger: When a code task runs, the system checks whether the task run updates the specified check scope of the current table. This is suitable for tables whose modification tasks are not fixed or for key tables that require monitoring for every change. Note We recommend that you select the task-updated partition as the check scope. For non-partitioned tables, the entire table is checked. The system automatically identifies all data changes and performs checks to avoid omissions. Fixed Task Trigger: Executes the configured quality rule after or before a specified task runs successfully. You can select tasks of the following types to trigger the rule: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and Database SQL nodes. This is suitable for scenarios where table modification tasks are fixed. Note Fixed task triggers can select only tasks in the production environment. If you configure a strong rule and its scheduled check fails, it may affect production tasks. Proceed with caution. The supported engine types are MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, AsiaInfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp TDH 6.x, and Transwarp TDH 9.3.x. Trigger time: Select when the quality check is triggered. You can choose to Trigger After All Tasks Succeed, Trigger After Each Task Succeeds, or Trigger Before Each Task Runs. Triggering task: The following roles can select task nodes in a production project to trigger the task. You can also search by node output name. Project administrators of Prod/Basic projects, users with the O&M system role for Prod projects, users with the Developer system role for Basic projects, and users with a custom project role that has Project Quality Management-Quality Rule Management permissions in Prod/Basic projects can select task nodes in the production project. Users with a custom global role that has Quality Rules-Manage permissions in Prod/Basic projects can select task nodes in all production projects. Note If you set Trigger Time to Trigger After All Tasks Succeed, we recommend that you select tasks with the same schedule period to avoid delays in rule execution and quality check results.
   Scheduling condition	Disabled by default. If enabled, the system first checks the scheduling condition. The quality rule is scheduled only if the condition is met; otherwise, the system skips the current schedule. Business Date/Execution Date: If you select Time-based Schedule (which does not support Execution Date), Data Update Trigger, or Fixed Task Trigger as the schedule type, you can configure a date condition. You can select a General Calendar or a Custom Calendar. For information about how to customize a calendar, see Create a public calendar. If you select General Calendar, the available conditions are Month, Day of Week, and Date. Example: If you select Custom Calendar, the available conditions are Date Type and Tag. Example: Instance type: If you select Data Update Trigger or Fixed Task Trigger as the schedule type, you can configure the instance type. You can select Scheduled instance, Data backfill instance, or Manually triggered instance. Example: Note You must configure at least one rule. To add a rule, click +Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.
   Check scope	If the schedule type is Time-based Schedule or Fixed Task Trigger, the check scope can be a Custom Check Scope. If the schedule type is Data Update Trigger, the check scope can be Task-updated Partition or Custom Check Scope. Task-updated Partition: If a partition is updated during the check task, the task runs directly on the updated partition. Note In dynamic partition scenarios, partitions may not be resolved, and no quality check will be performed. Fluctuation check rules, such as checking partition size, row count, or field statistics, require a specific partition and do not support the task-updated partition check scope. For non-partitioned tables, the entire table is checked if there are data updates. Custom Check Scope: For scenarios where partitions cannot be resolved, you can use a custom check scope and specify the check scope expression based on the business date or execution date. Check scope expression: An editable drop-down box where you can either enter the desired check scope, such as ds='${yyyyMMdd}', or select a built-in partition expression and modify it to quickly configure the setting. For more information about partition expressions, see Built-in partition expression types. Note If you have multiple conditions, you can connect them with and or or. For example, province="Zhejiang" and ds<=${yyyyMMdd}. If a filter condition is configured in the quality rule, the system combines it with the check scope expression using an AND operator. Both conditions are then used for data validation filtering. The check scope expression supports full table scans. Note: A full table scan consumes significant resources and is unsupported in some scenarios. We recommend that you configure a partition expression to avoid full table scans. Check Scope Preview: Defaults to the current business date.

3. Click OK to complete the schedule configuration.

Schedule configuration list

After a schedule is created, you can view, edit, clone, or delete it from the schedule configuration list.

Alert configuration

You can configure different alert methods for different rules to differentiate them. For example, you can configure phone call alerts for strong rule exceptions and SMS alerts for weak rule exceptions. If a rule matches multiple alert configurations, you can set a policy to determine which alert takes effect.

1. On the Quality Rule Details page, click the Alert Configuration tab, and then click Create Alert Configuration to open the Create Alert Configuration dialog box.

   

2. In the Create Alert Configuration dialog box, configure the parameters.

   Parameter	Description
   Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom. Note For a single monitored object, you can configure one alert for each of the three scopes: All Rules, All Strong Rules, and All Weak Rules. Newly added rules automatically use the corresponding alert configuration based on their severity. To change one of these alert configurations, you can modify the existing configuration. For a custom scope, you can select from all configured rules under the current monitored object, up to a maximum of 200 rules.
   Alert configuration name	The alert configuration name must be unique within a single monitored object and can be up to 256 characters long.
   Alert recipient	Configure the alert recipients and alert methods. You must select at least one alert recipient and one alert method. Alert recipient: You can select from three types of alert recipients: Custom, On-call Schedule, and Quality Owner. You can configure up to 5 custom alert recipients and up to 3 on-call schedules. Alert method: You can select from different notification methods, including Phone, Email, SMS, DingTalk, Feishu, WeCom, and Custom Channel. This notification method can be managed through channel settings.

3. Click OK to complete the alert configuration.

Alert configuration list

After you configure an alert, you can sort, edit, or delete it from the alert configuration list.

Quality report

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

• You can quickly filter check details by exception result, partition time, or rule or object name keyword.

• In the Operations column of the rule check details list, click the icon to view the rule check details for the quality rule.

• In the Operations column of the rule check details list, click the icon to view the execution log for the quality rule.

Configure permission management

1. Click Permission Management and configure Details Viewable By to specify which members can view check record details, quality rule details, and quality reports.

   Details Viewable By: You can select All Members or Only members with quality management permissions for the current object.

2. Click OK to complete the permission management configuration.

Configure runtime settings

1. Click the Runtime Configuration tab, then click the Edit button at the bottom to configure the parameters.

   Parameter	Description
   Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom.
   Run timeout	If the total runtime of a quality rule task (from start to finish, excluding resource and scheduling wait times) exceeds the set threshold without completing, the system automatically terminates it and marks it as failed. The system default is 1 hour. You can customize the duration within a range of 0 to 2 hours, with precision up to two decimal places.
   Auto-retry on failure	For failed quality rule tasks, you can decide whether to rerun the task based on the Retry Attempts and Retry Interval. This is disabled by default. Enable it to configure the settings. Retry Attempts: The maximum number of times a failed quality rule task will be automatically retried. The default is 1, and you can configure an integer between 1 and 3. Retry Interval: The time interval between each automatic retry. The default is 0 minutes, and you can configure an integer between 0 and 10 minutes.

2. Click OK to complete the runtime configuration.

Next steps

After you configure the quality rule, you can view it on the metric rule list page. For more information, see View the monitored object list.