Set Up Quality Rules for Real-Time Metadata Tables - Dataphin - Alibaba Cloud - Dataphin

This topic describes how to create a quality rule for a real-time meta table.

Prerequisites

You can configure quality rules only after you add a monitoring object. For more information about how to add a monitoring object, see Add a monitoring object.

Permissions

A super administrator, quality administrator, a user with a custom global role that has the Quality Rules-Manage permission, a user with a custom project role that has the Project Quality Management-Quality Rule Management permission on the meta table's project, and the owner of the real-time meta table can configure scheduling, alerts, and runtime configurations for quality rules.
A quality owner or a regular user must also have synchronous read permissions on the real-time meta table. To apply for this permission, see Apply for, renew, and revoke table permissions.
The supported operations vary depending on the object. For more information, see Permissions on quality rule operations.

Dry run vs. run

A dry run and a run differ in their execution methods and result display. A dry run is a simulated execution of a quality rule to verify its correctness and performance, and its results do not appear in the quality report. A run is an actual execution of a quality rule at a specified time, and its results are included in the quality report.

Meta table rules

Rule type	Description
Statistical trend monitoring	Validates data values and data change trends.
Real-time multi-link comparison	In scenarios that require strong business continuity, you can use real-time dual-link or triple-link quality rules to monitor data. If an anomaly occurs, operations staff can promptly switch to a backup data source or back up data. Real-time multi-link comparison quality rules can monitor issues such as data backlogs and statistical deviations.
Real-time and offline data reconciliation	When real-time and offline data use the same statistical logic, this quality rule can detect discrepancies between them. A significant difference may indicate a data quality issue.

Quality rule configuration

On the Dataphin homepage, choose Governance > Data Quality in the top navigation bar.
In the left-side navigation pane, click Quality Rules. On the Real-time Meta Table page, click the name of your target object to open the Quality Rule Details page to configure quality rules.
On the Quality Rule Details page, click Create Quality Rule.

In the Create Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic Information	Rule name	Enter a name for the quality rule.
	Rule strength	You can select weak rule or strong rule. If you select weak rule, an alert is triggered when the quality check fails, but downstream task nodes are not blocked. If you select strong rule, an alert is triggered when the quality check fails. If there are downstream tasks (triggered by code check scheduling or task-triggered scheduling), the system blocks the tasks to prevent the spread of corrupted data. If there are no downstream tasks (such as with periodic quality scheduling), only an alert is triggered.
	Description	Enter a description for the quality rule. The description can be up to 128 characters long.
	Rule template	You can select a Consistency or Stability rule template. Consistency: Includes Real-time and offline comparison and Real-time link comparison. Stability: Includes Real-time statistical value check. For more information, see Real-time meta table template types.
	Rule type	The rule type is associated with the template and is a basic attribute used for description and filtering.
Rule configuration	Rule configuration	Configure the rule based on the selected Rule Template. For more information, see Configure offline link comparison parameters and Configure multi-link comparison parameters.
Validation configuration	Rule validation	After the system validates a data quality rule, it compares the result with the exception validation settings. If the result meets the exception conditions, the validation fails, and the system triggers subsequent processes such as alerting. The available metrics for exception validation are determined by the template and its configuration. You can use AND/OR logic for multiple conditions, but we recommend using no more than three conditions. For more information, see Validation configuration details.
Business attribute configuration	Attribute information	The input format for business attributes depends on the quality rule attribute configuration. For example, if the value type for the governing department field is an enumeration (multi-select) with options such as Big Data Department, Business Department, and Technology Department, then this attribute will be a multi-select drop-down list with these options when you create a quality rule. If the value type for the rule owner field is custom input with a maximum length of 256 characters, you can enter a string up to 256 characters long for this attribute. If the input method for an attribute field is a Range, configure it as follows: Range: This is typically used for continuous numerical or date ranges. You can use the operators >, >=, <, and <=. For more information about attribute configuration, see Create and manage quality rule properties.
Scheduling attribute configuration	Scheduling type	You can select a pre-configured schedule. If you have not yet decided on a scheduling type, you can configure it after creating the quality rule. To create a new schedule, see Create a schedule.

Click OK to complete the quality rule configuration.
You can click Preview SQL to compare the SQL from the current configuration with the SQL from the last saved configuration. This helps you review SQL changes.
Note
- The Preview SQL button is disabled if required information is missing.
- The left pane shows the SQL preview of the last saved configuration. If no configuration was saved, this pane is empty. The right pane shows the SQL preview of the current configuration.

Rule configuration list

On the rule configuration list page, you can view the configured meta table rules and perform operations such as viewing, editing, dry running, running, and deleting.

Area	Description
① Filter and search area	You can quickly search by object or rule name. You can filter by rule type, rule template, rule strength, dry run status, and effective status. Note If a business attribute is configured to be searchable or filterable and is enabled, you can search or filter by that attribute.
② List area	Displays information about the configured rules, including Object Type/Name, Rule Name/ID, Dry Run Status, Effective Status, Rule type, Rule template, Rule strength, Scheduling Type, and related knowledge base document information. Click the icon before the refresh button to select the columns you want to display in the rule list. Effective status: We recommend performing a dry run before enabling a rule to prevent faulty rules from blocking production tasks. When a rule is enabled, it runs automatically according to its configured schedule. When a rule is disabled, it does not run automatically, but you can still run it manually. Related knowledge base document: Click View Details to see information from the knowledge base associated with the rule. This includes the table name, check object, rule, and related knowledge base documents. You can also search, view, edit, or delete documents in the knowledge base. For more information, see View the knowledge base.
③ Operations area	You can perform the following operations: View, Clone, Edit, Dry Run, Run, Scheduling Configuration, Associate Knowledge Base Document, and Delete. View: View the rule configuration details. Clone: Quickly clone a rule. Edit: After editing a rule, you must perform another dry run. Dry Run: After a dry run, you can click the icon to View dry run log. Run: After running a rule, you can view the validation results in Validation Records. Scheduling Configuration: In the dialog box that appears, you can filter by schedule type or search by schedule name. You can also edit the schedule. Associate Knowledge Base Document: After a rule is associated with a knowledge base document, you can view the associated information in Quality Rules and Governance Workbench. You can select a knowledge base that is not yet associated. To create a knowledge base, see Create and manage a knowledge base. Delete: Deleting a quality rule object deletes all quality rules under it. This action cannot be undone.
④ Batch operations area	You can perform the following operations in batches: Dry Run, Run, Configure Scheduling, Enable, Disable, Modify Business Attributes, Associate Knowledge Base Document, and Delete. Dry Run: Run dry runs for multiple rules in a batch. After the dry runs are complete, you can click the icon to View dry run log. Run: Run multiple rules in a batch. After they have run, you can view the validation results in Validation Records. Scheduling Configuration: Configure schedules for multiple rules in a batch. In the dialog box, you can filter by schedule type or search by schedule name. You can also edit schedules. You can only modify rules that are editable on the Quality Rules list page. Enable: Enable multiple rules in a batch. The selected rules will run automatically according to their schedules. You can only enable rules that are editable on the Quality Rules list page. Disable: Disable multiple rules in a batch. The selected rules will not run automatically, but you can still run them manually. You can only disable rules that are editable on the Quality Rules list page. Modify Business Attributes: You can batch modify business attributes if the corresponding field's value type is single-select or multi-select. If the value type is multi-select, you can append or modify attribute values. If the value type is single-select, you can directly modify the attribute value. Associate Knowledge Base Document: Associate multiple monitoring objects with a knowledge base in a batch. After a rule is associated, you can view the related information in Quality Rules and Governance Workbench. To create a knowledge base, see Create and manage a knowledge base. Delete: Delete multiple quality rule objects in a batch. This action cannot be undone, so proceed with caution. You can only delete rules that are editable on the Quality Rules list page.

Create a schedule

Note

When you configure a schedule for a rule, you can select from the existing schedules for the current table. A maximum of 20 schedules can be created for each table.
A single rule can be associated with a maximum of 10 schedules.
Duplicate schedules with identical configurations are automatically removed.

On the Quality Rule Details page, click the Scheduling Configuration tab, and then click the Create Scheduling button. The Create Scheduling dialog box opens.

In the Create Schedule dialog box, configure the parameters.

Parameter	Description
Schedule name	Enter a name for the schedule.
Scheduling type	You can select time-based scheduling, scheduling triggered by data update, or scheduling triggered by a specified task. Time-based scheduling: Runs quality checks on a fixed, recurring schedule. This is suitable for scenarios where data is generated at predictable times. Scheduling Cycle: Running quality rules consumes computing resources. We recommend avoiding running multiple quality rules at the same time to prevent impacting production tasks. You can set the scheduling cycle to Day, Week, Month, Hour, or Minute. If the system time zone (the time zone in your user center) is different from the scheduling time zone (the time zone configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Scheduling triggered by data update: When any code task runs, the system checks if the run updates the specified validation scope of the current table. This is suitable for tables that are modified by various tasks or for critical tables that require monitoring after every change. Note We recommend setting the validation scope to the partitions updated by the task. For non-partitioned tables, the entire table is validated. The system automatically detects and validates all data changes to avoid omissions. Scheduling triggered by a specified task: Runs the configured quality rule after a specified task completes successfully or before the task starts. You can select tasks of the following engine types as triggers: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and database SQL nodes. This is suitable for scenarios where the table is modified by a fixed task. Note You can only select production tasks as triggers. If the rule is a strong rule, a validation failure might impact online tasks. Use this option with caution. Supported engines include MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, Asiainfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp Data Hub (TDH) 6.x, and TDH 9.3.x. Trigger time: Select when to trigger the quality check. Options include Trigger after all tasks run successfully, Trigger each time a task runs successfully, and Trigger before each run of a task. Trigger task: The following roles can select a task node from a production project as a trigger. You can also search by the output name of the node. Project administrators for Prod/Basic projects, users with the Ops system role for Prod projects, users with the Developer system role for Basic projects, and users with a custom project role that has the Project Quality Management-Quality Rule Management permission in a Prod/Basic project can select task nodes from production projects. Users with a custom global role that has the Quality Rules-Manage permission can select task nodes from all production projects. Note If you select Trigger after all tasks run successfully, we recommend selecting trigger tasks that have the same scheduling cycle as the quality rule. This prevents delays in rule execution and results.
Scheduling conditions	This feature is disabled by default. If enabled, the system checks if the scheduling conditions are met before running the quality rule. The rule runs only when the conditions are met. Otherwise, the scheduled run is skipped. Business Date/Execution Date: If the schedule type is time-based scheduling (which does not support Execution Date), scheduling triggered by data update, or scheduling triggered by a specified task, you can configure date conditions. You can select General calendar or Custom calendar. For information about how to create a custom calendar, see Create a public calendar. If you select General calendar, the conditions can be Month, Day of the week, or Date. See the following figure: If you select Custom calendar, the conditions can be Date Type or Tag. See the following figure: Instance type: If the schedule type is scheduling triggered by data update or scheduling triggered by a specified task, you can configure the instance type. Options include Periodic instance, Backfill instance, and Manual instance. See the following figure: Note You must configure at least one rule. To add a rule, click + Add rule. You can configure a maximum of 10 scheduling conditions. You can set the relationship between scheduling conditions to AND or OR.

Click OK to save the schedule configuration.

Scheduling configuration list

After a schedule is created, you can view, edit, clone, and delete it from the scheduling configuration list.

Area	Description
① Filter and search area	You can search by schedule name. You can filter by time-based scheduling, scheduling triggered by data update, or scheduling triggered by a specified task.
② List area	Displays the Schedule Name, Scheduling Type, Last Updater, and Last Update Time for the configured rules.
③ Operations area	You can edit, clone, or delete a schedule. Edit: Modify a configured schedule. Important Changes will be synchronized to all rules that reference this schedule. Proceed with caution. Clone: Quickly copy a schedule configuration. Delete: You cannot delete a schedule that is referenced by a rule.

Alert configuration

You can configure different alerting methods for different rules to differentiate alerts. For example, you can configure phone call alerts for strong rule failures and SMS alerts for weak rule failures. If a rule matches multiple alert configurations, you can define a policy to determine which configuration takes effect.

Note

You can create a maximum of 20 alert configurations for a single monitoring object.

On the Quality Rule Details page, click the Alert Configuration tab, and then click the Create Alert Configuration button to open the Create Alert Configuration dialog box.

In the Create Alert Configuration dialog box, configure the parameters.

Parameter	Description
Scope	You can select All rules, All strong rules, All weak rules, or Custom. Note For a single monitoring object, you can create one alert configuration for each of the following scopes: All rules, All strong rules, and All weak rules. New rules automatically match the corresponding alert configuration based on their strength. To change one of these configurations, edit the existing one. For a custom scope, you can select up to 200 existing rules under the current monitoring object.
Alert configuration name	The alert configuration name must be unique within a single monitoring object and can be up to 256 characters long.
Alert recipient	Configure alert recipients and methods. You must select at least one recipient and one method. Alert recipient: You can select Custom, on-call schedule, or quality owner. You can configure up to five custom alert recipients and up to three on-call schedules. Alerting method: You can select Phone, Email, SMS, DingTalk, Lark, WeCom, and Custom Channel as notification methods. These notification methods can be managed by using channel settings.

Click OK to save the alert configuration.

Alert configuration list

After an alert configuration is created, you can sort, edit, and delete it from the alert configuration list.

No.	Description
① Sorting area	Configure the alert policy for when a quality rule matches multiple alert configurations: The first matched alert configuration takes effect: With this policy, only the first matching alert configuration is applied. You can sort the alert configurations. Click Sort Rules, and then drag the configuration by using the icon, or use the icons in the Operations column to move it. The icons, from left to right, are Move to Top and Move to Bottom. After you have sorted the configurations, click Finish Sorting. All alert configurations take effect: All alert configurations in the list apply to the quality rules under the current monitoring object. For example, if you configure multiple alert configurations and select this policy, the system consolidates alerts by alerting method, alert recipient, and alert rule. Specifically, if the recipient is the same user and the alerting methods are Custom and Quality Owner, the alert messages are consolidated. Note Alert consolidation is not supported for on-call schedules. If a quality owner is a user group, alerts are automatically sent to all members of the group.
② List area	Displays the name, effective scope, specific recipients for each alert type, and their corresponding alerting methods. Effective scope: For custom alerts, you can view the configured object name and rule name. If a rule is deleted, the object name is no longer displayed, and we recommend updating the alert configuration.
③ Operations area	You can edit and delete alert configurations. Edit: Modify a configured alert. If you change the recipients or alerting method, promptly notify the relevant personnel to avoid missing business alerts. Delete: When you delete a configuration, the associated rules will no longer trigger alerts. Proceed with caution.

View the quality report

Click Quality Report to view the Rule Validation Overview and Rule Validation Details for the current quality rule.

You can filter the validation details by exception result, partition time, or rule/object name keyword.
In the Operations column of the rule validation details list, click the icon to view the validation details for the quality rule.
In the Operations column of the rule validation details list, click the icon to view the execution log for the quality rule.

Quality rule permission management

Click Permission Management and configure who can view validation record details, quality rule details, and quality reports under Viewable details.
Viewable details: You can select All members or Only members with quality management permissions on the current object.
Click OK to save the permission configuration.

Quality rule runtime configuration

Note

If a quality rule task is automatically terminated due to a timeout, its status is set to failed. You can view the corresponding execution log on the validation record details page.
If a quality rule is configured to be retried on failure, you can view the retry attempt records in the execution log on the validation record details page.
For a single monitoring object, all quality rules (including new and existing rules that have not yet run) use the latest runtime configuration.

Click the Runtime Configuration tab, click Edit at the bottom, and configure the parameters.

Parameter	Description
Scope	You can select All rules, All strong rules, All weak rules, or Custom.
Run timeout	If a quality rule task runs longer than the specified threshold, the system automatically terminates it and marks it as failed. The duration does not include resource or scheduling wait times. The system default is 1 hour. You can set a custom duration from 0 to 2 hours, with a precision of two decimal places.
Automatic retries on failure	If a quality rule task fails, you can configure it to be retried based on the Number of retries and Interval. This feature is disabled by default. Number of retries: The maximum number of times a failed task can be automatically retried. The default is 1. You can set an integer from 1 to 3. Interval: The time interval between each automatic retry. The default is 0 minutes. You can set an integer from 0 to 10 minutes.

Click OK to save the runtime configuration.

Next steps

After you configure the quality rules, you can view them on the real-time meta table rule list page. For more information, see View the list of monitoring objects.