Dataphin:Create a metric quality rule

Prerequisites

Before you configure a quality rule, you must add a monitoring object. For more information, see Add a monitoring object.

Permissions

• A super admin, quality admin, a user with a custom global role that has Quality Rules-Manage permissions, a user with a custom project role that has Project Quality Management-Quality Rule Management permissions for the table's project, and the metric business owner can configure schedules, alerts, and runtime configurations for quality rules.

• A quality owner or a standard user must also have read permissions on the logical table fields. To apply for these permissions, see Apply for, renew, and return table permissions.

• The supported operation permissions vary by object. For more information, see Quality rule operation permissions.

How checks work

When a quality rule check runs for a metric, if a weak rule is triggered, the system sends you an alert. This alert helps you quickly identify and resolve the issue. If a strong rule is triggered, the system sends an alert and automatically interrupts the associated task to prevent dirty data from flowing downstream.

Trial run vs. run

A trial run and a run differ in their execution method and where their results are displayed. A trial run executes a quality rule once in simulation mode to test its correctness and runtime behavior. Trial run results are not displayed in the quality report. A run executes a quality rule at a specific time, and the results appear in the quality report for you to review.

Quality rule configuration

1. In the top navigation bar of the Dataphin homepage, choose Governance > Data Quality.

2. In the left-side navigation pane, click Quality Rules. On the Metrics page, click the name of your target object to go to the Quality Rule Details page and configure the quality rule.

3. On the Quality Rule Details page, click Create Quality Rule.

4. In the Create Quality Rule dialog box, configure the parameters.

   Parameter		Description
   Basic information	Rule name	Enter a custom name for the quality rule.
   	Rule strength	Select weak rule or strong rule. If you select weak rule, an alert is triggered when the quality rule check fails, but downstream task nodes are not blocked. If you select strong rule, an alert is triggered when the quality rule check fails. If there are downstream tasks (in code check scheduling or task-triggered scheduling), the downstream tasks are blocked to prevent dirty data from flowing downstream. If there are no downstream tasks (such as in periodic quality scheduling), only an alert is sent.
   	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
   	Configuration method	Create from template: Quickly create quality rules from general system templates or custom business templates. System template: Suitable for creating general rules. Built-in parameters are configurable. Custom template: Typically used for rules that contain business logic. Parameters are preset and do not require configuration. Custom SQL: Define quality monitoring rules by using SQL for maximum flexibility. This method is ideal for complex scenarios.
   	Rule template	Select a rule template from the drop-down list. Options include uniqueness, stability, and custom SQL. Uniqueness: Includes Field Group Count Check and Duplicate Field Value Count Check. Stability: Includes Field Stability Check and Field Fluctuation Check. Custom SQL: Includes Custom Statistical Metric Check. For more information, see Quality rule template types.
   	Rule type	The rule type is associated with the template and serves as its most basic attribute. It can be used for descriptions and filtering.
   Rule configuration	Rule configuration	If you select uniqueness for Rule template, the following parameters are available. Field Group Count Check/Duplicate Field Value Count Check: Filter Data of Table to Check: This option is disabled by default. If you enable it, you can configure filter conditions for the table, such as partition filters or standard row filters. These conditions are appended to the check SQL. If the table requires a partition filter, configure a partition expression in the scheduling configuration. This lets you view the quality report at the partition granularity. Enter the filter condition. Examples: id = 12 -- for a single table T1.id=12 and T2.name = "John" -- for two tables If you select stability for Rule template, the following parameters are available. Field Stability Check/Field Fluctuation Check: Statistical Method: Select a statistical method based on your business scenario. Filter Data of Table to Check: This option is disabled by default. If you enable it, you can configure filter conditions for the table, such as partition filters or standard row filters. These conditions are appended to the check SQL. If the table requires a partition filter, configure a partition expression in the scheduling configuration. This lets you view the quality report at the partition granularity. Enter the filter condition. Examples: id = 12 -- for a single table T1.id=12 and T2.name = "John" -- for two tables If you select custom SQL for Rule template, the following parameters are available. Custom Statistical Metric Check: Custom SQL: Only SELECT queries are supported. The query object must include the primary table. Example: select sum(sale) from tableA where ds=${bizdate};
   Verification configuration	Rule verification	After a quality rule check completes, the system compares the result with the exception check configuration. If the exception conditions are met, the check fails and triggers subsequent processes, such as alerts. The available metrics for an exception check depend on the template and its configuration. Multiple AND/OR conditions are supported, but we recommend using fewer than three conditions in practice. For more information, see Metric check configuration.
   Business property configuration	Property information	Business property input rules depend on how the quality rule properties are configured. Examples: For example, if an attribute for the managing department is a multi-choice enum with the values Big Data Department, Business Department, and Technology Department, it appears as a multi-select drop-down list when you create a quality rule. For the attribute field that maps to the rule owner, the value type is custom input and the maximum length is 256. Therefore, when you create a quality rule, you can enter up to 256 characters. If the attribute field input method is Range, configure it as follows: Range: This is commonly used when values are continuous numbers or dates. You can select >, >=, <, or <=. For more information, see Create and manage quality rule attributes.
   Scheduling property configuration	Scheduling mode	Select an existing schedule. If you have not decided on a scheduling mode, you can create the quality rule first and configure it later. To create a schedule, see Create a schedule.

5. Click Save to complete the rule configuration.

   You can click Preview SQL to compare this configuration with the last saved configuration and review SQL changes.

   Note

   • The Preview SQL button is unavailable if key information is not complete.

   • The left pane shows the SQL from the last saved configuration, or is empty if none was saved. The right pane shows the SQL preview of the current configuration.

   Quality rule list

   On the rule configuration list page, you can view information about the configured metric rules and perform operations such as view, edit, trial run, run, and delete.

   

   Area	Description
   ① Filter and search area	You can search by object or rule name. You can filter by rule type, rule template, rule strength, trial run status, or effective status. Note If a quality rule attribute is configured to be searchable or filterable and is enabled, you can search or filter by that attribute.
   ② List area	This area displays the object type/name, rule name/ID, trial run status, effective status, rule type, rule template, rule strength, scheduling type, and related knowledge base document information for each rule. Click the icon next to the refresh button to select which columns to display in the rule list. Effective status: We recommend that you perform a trial run before you activate a rule. Enable the effective status only for rules that pass the trial run. This prevents incorrect rules from blocking production tasks. After a rule is enabled, it automatically runs according to its schedule. After a rule is disabled, it no longer runs automatically, but you can still run it manually. Related Knowledge Base Documents: Click View Details to view information about the knowledge base associated with the rule. This information includes the table name, validation object, rule, and related knowledge base documents. You can also search for, view, edit, or delete these documents. For more information, see View the knowledge base.
   ③ Actions area	You can perform the following operations: View, Clone, Edit, Trial Run, Run, Scheduling Configuration, Associate Knowledge Base Document, and Delete. View: View the rule configuration details. Clone: You can quickly clone rules. Edit: After you edit the rule, you must perform a trial run again. Trial Run: You can perform a trial run based on an Existing Schedule or a Custom Verification Scope. After the run, you can click the icon to View Trial Run Logs. Run: You can run the rule by selecting an Existing schedule or a Custom verification scope. After the rule is run, you can view the verification results in Verification Records. Scheduling configuration: You can filter by scheduling type or search by schedule name in the dialog box. You can also edit schedules. Associate Knowledge Base Document: After you associate a rule with a knowledge base, you can view the associated knowledge in the quality rule and the Governance Workbench. You can select a knowledge base that is not already associated. To create a knowledge base, see Create and manage knowledge bases. Delete: Deleting this quality rule object also deletes all quality rules under it. This action cannot be undone. Proceed with caution.
   ④ Batch operations area	You can perform the following operations in bulk: Trial Run, Run, Configure Schedule, Enable, Disable, Modify Business Properties, Associate Knowledge Base Document, and Delete. Trial run: You can perform a trial run of rules in batches by selecting an Existing schedule or a Custom verification scope. After the trial run, click the icon to View trial run logs. Run: Runs rules in batches based on an existing schedule or a custom verification scope. After the run, you can view the verification results in Verification Records. Scheduling configuration: You can filter by scheduling type or search for schedules by name in the dialog box. You can also edit schedules and configure schedules for quality rules in batches. You can modify only the selected rules that are editable on the quality rules list page. Enable: After you batch enable the effective status, the selected rules are automatically executed according to their configured schedules. You can enable only the selected rules that are editable on the Quality Rules list page. Disable: After you disable the selected rules in a batch, they are not automatically executed. However, you can execute them manually. You can disable only the selected rules that are editable on the Quality Rules list page. Modify Business Attributes: You can modify business attributes in batches when the value type of the corresponding field is single-select or multi-select. If the field type is multi-select, you can append or modify property values. If the field type is single-select, you can directly modify the property value. Associated knowledge base documents: After you associate a rule with knowledge, you can view the associated knowledge in the quality rules and the governance workbench. You can configure knowledge bases for monitoring objects in batches. To create a knowledge base, see Create and manage a knowledge base. Delete: Deletes the selected quality rules that are editable on the Quality Rules page. This action cannot be undone. Proceed with caution.

Create schedule

1. On the Quality Rule Details page, click the Scheduling Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

2. In the Create Schedule dialog box, configure the parameters.

   Parameter	Description
   Schedule name	Enter a custom name for the schedule.
   Scheduling type	The available types are timed scheduling, data update-triggered scheduling, and fixed task-triggered scheduling. Timed scheduling: Performs data quality checks at scheduled times or intervals. This is suitable for scenarios where data is generated at relatively fixed times. Scheduling period: Running quality rules consumes compute resources. We recommend that you avoid running multiple quality rules concurrently to prevent affecting production tasks. The scheduling period options are Day, Week, Month, Hour, and Minute. If the system time zone (the user's time zone) is different from the scheduling time zone (configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Data update-triggered scheduling: When any code task runs, the system checks whether the task updates the specified verification scope of the current table. This is suitable for tables that are modified at irregular intervals or critical tables that require monitoring after every change. Note We recommend selecting partitions updated by task as the verification scope. For non-partitioned tables, the entire table is checked. The system automatically identifies all data changes and performs checks, preventing any omissions. Fixed task-triggered scheduling: Executes the configured quality rule after a specified task runs successfully or before it runs. You can select tasks of the following node types to trigger the rule: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and Database SQL. This is suitable for tables that are modified by predictable, recurring tasks. Note You can select only production environment tasks for fixed task-triggered scheduling. If the rule strength is set to strong, a check failure might affect production tasks. Proceed with caution. Supported engine types include MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, AsiaInfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp TDH 6.x, and Transwarp TDH 9.3.x. Trigger timing: Select when the quality check is triggered. Options include Trigger after all tasks run successfully, Trigger after each task runs successfully, and Trigger before each task runs. Trigger task: The following roles can select task nodes from production projects to trigger the task. You can also search by node output name. Project administrators of Prod/Basic projects, users with the O&M system role in Prod projects, users with the Developer system role in Basic projects, and users with a custom project role that has Project Quality Management-Quality Rule Management permissions in Prod/Basic projects can select task nodes from production projects. Users with a custom global role that has Quality Rules-Manage permissions can select task nodes from all production projects. Note When the trigger timing is set to Trigger after all tasks run successfully, select tasks with the same scheduling period to avoid delays in rule execution and result generation.
   Scheduling conditions	This is disabled by default. If enabled, the system checks whether the scheduling conditions are met before the quality rule is scheduled. The schedule runs only if the conditions are met; otherwise, it is skipped. Business date/Execution date: If the scheduling type is timed scheduling (does not support Execution Date), data update-triggered scheduling, or fixed task-triggered scheduling, you can configure a date condition. You can select Common Calendar or Custom Calendar. For information on how to create a custom calendar, see Create a public calendar. If you select Common Calendar, you can set conditions for Month, Day of Week, and Date. Example: If you select Custom Calendar, you can set conditions for Date Type and Tag. Example: Instance type: If the scheduling type is data update-triggered scheduling or fixed task-triggered scheduling, you can configure the instance type. Options include periodic instance, backfill instance, and manual instance. Example: Note You must configure at least one rule. To add a rule, click + Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.
   Verification scope	When the scheduling type is timed scheduling or fixed task-triggered scheduling, the supported verification scope is custom verification scope. When the scheduling type is data update-triggered scheduling, the supported verification scopes are partitions updated by task and custom verification scope. Task update partitions: If a check task updates partitions, the task is dispatched directly to the updated partitions. Note In dynamic partition scenarios, partitions may not be resolved, and no quality check will be performed. Because volatility check rules (such as those for partition size, row count, or field statistics) require a specified partition, they do not support the "Partitions Updated by Task" verification scope. If a non-partitioned table is updated, the entire table is checked. Custom verification scope: For scenarios where partitions cannot be resolved, you can use a custom verification scope and specify a verification scope expression based on the business date or execution date. Verification scope expression: This is an editable dropdown list. You can directly enter a verification scope, such as ds='${yyyyMMdd}'. You can also select a built-in verification scope expression and then modify it to quickly complete the configuration. For more information about partition expressions, see Built-in partition expression types. Note If you have multiple conditions to validate, you can connect them with and or or. For example, province="Zhejiang" and ds<=${yyyyMMdd}. If a filter condition is configured in the quality rule, it is combined with the verification scope expression by using an AND relationship. Both conditions are used to filter data during the check. The verification scope expression supports full table scans. Note: A full table scan consumes significant resources and is not supported by all engines. Configure a partition expression to avoid full table scans. Verification Scope Budget: Defaults to the current business date.

3. Click OK to save the scheduling configuration.

Scheduling configuration list

After a schedule is created, you can view, edit, clone, or delete it from the scheduling configuration list.

Alert configuration

You can configure different alert methods for different rules to differentiate alerts. For example, you can configure phone call alerts for strong rule exceptions and SMS alerts for weak rule exceptions. If a rule matches multiple alert configurations, you can set a policy to determine which alert takes effect.

1. On the Quality Rule Details page, click the Alert Configuration tab, and then click Create Alert Configuration to open the Create Alert Configuration dialog box.

   

2. In the Create Alert Configuration dialog box, configure the parameters.

   Parameter	Description
   Scope	You can select All rules, All strong rules, All weak rules, or Custom. Note For a single monitoring object, you can configure one alert each for the scopes "All Rules," "All Strong Rules," and "All Weak Rules." New rules automatically match the corresponding alert based on their strength. To change one of these alert configurations, you must modify the existing one. For a custom scope, you can select up to 200 of the rules that are already configured for the current monitoring object.
   Alert configuration name	The alert configuration name must be unique within a single monitoring object and can be up to 256 characters long.
   Alert recipient	Configure the alert recipients and alert methods. You must select at least one recipient and one method. Alert recipient: You can select Custom, on-duty schedule, or quality owner as the recipient type. You can configure up to 5 custom alert recipients and up to 3 on-duty schedules. Alert method: You can select from different notification methods, including Phone, Email, SMS, DingTalk, Lark, WeCom, and Custom Channel. These methods can be managed by configuring channel settings.

3. Click OK to save the alert configuration.

Alert configuration list

After an alert configuration is created, you can sort, edit, or delete it from the alert configuration list.

Quality reports

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

• You can filter the check details by exception result, partition time, or rule or object name keywords.

• In the Actions column of the rule check details list, click the icon to view the rule check details for the quality rule.

• In the Actions column of the rule check details list, click the icon to view the execution log for the quality rule.

Permission management

1. Click Permission management and configure View details permission to specify which members can view verification records, quality rule details, and quality reports.

   Viewable by: You can select All members or Only members with quality management permissions for the current object.

2. Click OK to save the permission configuration.

Runtime configuration

1. Click the Runtime configuration tab, and then click Edit at the bottom to configure the parameters.

   Parameter	Description
   Scope	You can select All rules, All strong rules, All weak rules, or Custom.
   Execution timeout	If the total runtime of a quality rule task (the time from start to finish, excluding resource and scheduling wait times) exceeds the specified threshold, the system automatically terminates the task and marks it as failed. The default is 1 hour. You can customize the duration within a range of 0 to 2 hours, with a precision of two decimal places.
   Automatic retry upon failure	For failed quality rule tasks, you can configure whether to automatically retry the task based on the number of retries and retry interval. This is disabled by default. Enable it to configure the settings. Number of retries: The maximum number of times a failed task can be automatically retried. The default is 1, and you can configure an integer from 1 to 3. Retry interval: The time interval between each automatic retry. The default is 0 minutes, and you can configure an integer from 0 to 10 minutes.

2. Click OK to save the runtime configuration.

Next steps

After you configure the quality rule, you can view it on the metric rules list page. For more information, see View the monitoring object list.