Set Up Data Quality Rules, Alerts & Connectivity Checks - Dataphin - Alibaba Cloud - Dataphin

Dataphin monitors your data source's connectivity and any changes to its table structure. A quality rule periodically checks for changes and lets you configure alerts for exceptions, keeping you informed of your data source's status in real time. This topic describes how to configure quality rules for a data source.

Prerequisites

You must add a monitored object before you can configure a quality rule. For more information, see Add a monitored object.

Limitations

You can create quality monitoring rules for any data source in Dataphin. While all supported data sources can be tested for connectivity, only some data sources support quality rules for table structure changes. For more information, see Data sources supported by Dataphin.

Permissions

Super administrators, quality administrators, users with a custom global role that grants Quality Rules-Manage permissions, and data source owners can configure schedules, alerts, and run configurations for quality rules.
Quality owners and regular users also need synchronous read permissions for the data source. To apply for data source permissions, see Apply for data source permissions.
Supported operation permissions vary by object. For more information, see Quality rule operation permissions.

Dry run vs. run

A dry run and a run differ in how they are executed and where their results are displayed. A dry run is a simulated execution of a quality rule to test its correctness and performance. The results of a dry run are not displayed in the quality report. A run is a scheduled check of a quality rule. A run outputs its results to the quality report.

Quality rule configuration

On the Dataphin homepage, choose Governance > Data Quality from the top menu bar.
In the left-side navigation pane, click Quality Rules. On the Data Source page, click the name of the target object to open the Quality Rule Details page.
On the Quality Rule Details page, click New Quality Rule.

In the New Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic Information	Rule Name	A custom name for the quality rule. The name can be up to 256 characters long.
	Rule Strength	You can set the strength to weak rule or strong rule. Weak rule: If you select Weak rule, an abnormal result from a quality rule check triggers an alert but does not block downstream task nodes. Strong rule: If you select Strong rule, an alert is triggered when a quality rule check fails. If there are downstream tasks (such as code check scheduling or task trigger scheduling), the tasks are also blocked to prevent the spread of polluted data. If there are no downstream tasks (such as periodic quality scheduling), only an alert is triggered.
	Description	A custom description for the quality rule. The description can be up to 128 characters long.
	Rule Template	Only Stability is supported, which includes Data source connectivity monitoring and Table structure change monitoring. Connectivity monitoring: Monitors data sources configured in Dataphin for connection failures caused by issues such as network changes or incorrect credentials (username or password), which can cause tasks to fail. You can configure monitoring and alerts for connectivity changes. Table structure change: Monitors upstream tables for structural changes, such as renaming, deleting, adding, or removing fields, which can cause downstream errors. You can configure monitoring and alerts for these changes.
	Rule Type	Derived from the template, the rule type is a basic attribute used for description and filtering.
	Rule Configuration
	Select Table to Validate	This parameter is available when you select the Table structure change monitoring rule template. Select the data table to validate based on its DB or schema.
Business Property Configuration	Property Information	Business property requirements depend on the underlying attribute definitions. For example: if the attribute for "managing department" is set as a multi-select enumeration with options like Big Data Department, Business Department, and Technology Department, this field will appear as a multi-select dropdown menu with those options when you create the rule. If the attribute for "rule owner" is set as custom input with a maximum length of 256 characters, you can enter a text string up to 256 characters long. If an attribute's input method is Range, configure it as follows: Range: Used for continuous numerical or date values. You can use the operators >, >=, <, and <=. For more information about configuring attributes, see Create and manage quality rule properties.
Schedule Property Configuration	Scheduling Method	Select a configured schedule. If you have not decided on a scheduling method, you can create the rule and configure it later. To create a new schedule, see Create a schedule.

Click OK to complete the rule configuration.

Rule configuration list

On the rule configuration list page, you can view information about configured data source rules and perform operations such as view, edit, dry run, run, and delete.

Area	Description
① Filter and Search Area	Allows you to quickly search by object or rule name. Allows you to filter by rule type, rule template, rule strength, dry run status, and activation status. Note If a business property is configured as searchable or filterable and is enabled, you can search or filter by that property.
② List Area	Displays rule configuration details, including object type/name, rule name/ID, dry run status, activation status, rule type, rule template, rule strength, schedule type, and related knowledge base information. Click the icon before the refresh button to customize the columns displayed in the list. Activation Status: We recommend performing a dry run before activating a rule. To avoid blocking production tasks with a faulty rule, activate it only after a successful dry run. When activated, the rule runs automatically based on its configured schedule. When deactivated, the rule no longer runs automatically but can still be run manually. Related Knowledge Base Documents: Click View Details to view the knowledge base information associated with the rule. This includes the table name, validation object, rule, and related documents. You can also search, view, edit, and delete knowledge base entries. For more information, see View a knowledge base.
③ Operations Area	You can perform operations such as view, clone, edit, dry run, run, configure schedule, associate knowledge base document, and delete. View: View the details of the rule configuration. Clone: Quickly clone a rule. Edit: After editing a rule, you must perform a new dry run. Dry Run: After a dry run, click the icon to View Dry Run Log. Run: After a run, you can view the results in the validation records. Schedule Configuration: In the pop-up dialog box, you can filter by schedule type or quickly search by schedule name. You can also edit schedules. Associate Knowledge Base Document: After a rule is associated with a knowledge base, you can view the related knowledge in Quality Rules and the governance workbench. You can select an unassociated knowledge base. To create one, see Create and manage a knowledge base. Delete: Deleting a quality rule object also deletes all quality rules under it. This action cannot be undone.
④ Batch Operations Area	You can perform batch operations such as dry run, run, configure schedule, activate, deactivate, modify business properties, associate knowledge base document, and delete. Dry Run: Perform a dry run on multiple rules at once. After the dry run, click the icon to View Dry Run Log. Run: Run multiple rules at once. After the run, you can view the results in validation records. Schedule Configuration: In the dialog box, you can filter by schedule type or search by schedule name. You can also edit schedules and apply schedule configurations to multiple quality rules in a batch. You can modify only selected rules that are editable. Activate: After you activate the selected rules, they run automatically according to their schedule configuration. You can activate only selected rules that are editable. Deactivate: After you deactivate the selected rules, they no longer run automatically but can be run manually. You can deactivate only selected rules that are editable. Modify Business Properties: You can batch modify business properties when their corresponding field value type is single-select or multi-select. If the value type is multi-select, you can append or modify property values. If the value type is single-select, you can directly modify property values. Associate Knowledge Base Document: After a rule is associated with a knowledge base, you can view the related knowledge in Quality Rules and the governance workbench. You can associate a knowledge base with multiple monitored objects in a batch. To create one, see Create and manage a knowledge base. Delete: You can batch delete quality rules. This action cannot be undone. Proceed with caution. You can delete only selected rules that are editable.

Create a schedule

Note

When you configure a schedule for a rule, you can quickly choose from existing schedules for the current table. Each table can have a maximum of 20 schedule rules.
You can configure a maximum of 10 schedules for a single rule.
Duplicate schedules with identical configurations are automatically removed.

On the Quality Rule Details page, click the Schedule Configuration tab, and then click New Schedule to open the New Schedule dialog box.

In the New Schedule dialog box, configure the parameters.

Parameter	Description
Schedule Name	A custom name for the schedule. The name can be up to 64 characters long.
Schedule Type	You can select Time-based schedule or Task-triggered schedule. Time-based schedule: Performs data quality checks at a scheduled time or on a recurring basis. This is suitable for scenarios where data is generated at relatively fixed times. Schedule Period: Running a quality rule consumes computing resources. We recommend that you avoid running multiple quality rules concurrently to prevent interference with production tasks. You can select a schedule period of Day, Week, Month, Hour, or Minute. If the system time zone (the time zone in your user center) is different from the schedule time zone (the time zone configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Task-triggered schedule: Runs the configured quality rule after a specified task runs successfully or before it runs. You can select tasks of the following engine types to trigger the rule: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and Database SQL nodes. This is suitable for scenarios with fixed table modification tasks. Note You can only select production environment tasks for task-triggered scheduling. If the rule strength is set to strong, a check failure in the scheduled task might affect online tasks. Proceed with caution based on your business needs. The supported engine types are MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, Asiainfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp TDH 6.x, and Transwarp TDH 9.3.x. Trigger Timing: Select when the quality check is triggered. You can choose to Trigger after all tasks run successfully, Trigger after each task runs successfully, or Trigger before each task runs. Trigger Task: The following roles can select a task node from a production project to serve as the trigger task. You can also search by node output name. Users with the Project Admin or O&M system role for a Prod project, users with the Developer system role for a Basic project, or users with a custom project role that grants Project Quality Management-Quality Rule Management permissions in a Prod/Basic project can select task nodes from the production project. Users with a custom global role that grants Quality Rules-Manage permissions in a Prod/Basic project can select task nodes from all production projects. Note If you select Trigger after all tasks run successfully, we recommend that you select trigger tasks with the same schedule period to avoid delayed rule execution and quality check results.
Scheduling Conditions	This feature is disabled by default. If enabled, the system checks if the scheduling conditions are met before running the quality rule. The rule runs only if the conditions are met. Otherwise, the current schedule is skipped. Business Date/Execution Date: If you select Time-based schedule (Execution Date not supported), code-check triggered schedule, or task-triggered schedule as the schedule type, you can set a condition on the date. You can select a General Calendar or Custom Calendar. For information about how to customize a calendar, see Create a public calendar. If you select General Calendar, you can set conditions for Month, Day of week, or Date. For example: If you select Custom Calendar, you can set conditions for Date Type or Tag. For example: Instance Type: If you select code-check triggered schedule or task-triggered schedule as the schedule type, you can set a condition on the instance type. You can select Periodic instance, Backfill instance, or Manual instance. For example: Note You must configure at least one condition. To add a condition, click + Add Condition. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.

Click OK to complete the schedule configuration.

Schedule configuration list

After a schedule is created, you can view, edit, clone, and delete it in the schedule configuration list.

Area	Description
① Filter and Search Area	Allows you to quickly search by schedule name. Allows you to filter by Time-based schedule or Task-triggered schedule.
② List Area	Displays the Schedule Name, Schedule Type, Last Updated By, and Last Updated Time.
③ Operations Area	You can edit, clone, or delete a schedule. Edit: Modify a configured schedule. Important All rule configurations that reference this schedule will be affected. Proceed with caution. Clone: Quickly copy a schedule configuration. Delete: Schedules that are referenced by a rule configuration cannot be deleted.

Configure alerts

You can configure different notification methods for different rules to differentiate their alerts. For example, you can configure phone alerts for strong rule exceptions and SMS alerts for weak rule exceptions. If a rule triggers multiple alert configurations, you can set a policy to determine which alert takes effect.

Note

You can create a maximum of 20 alert configurations for a single monitored object.

On the Quality Rule Details page, click the Alert Configuration tab, and then click New Alert Configuration to open the New Alert Configuration dialog box.

In the New Alert Configuration dialog box, configure the parameters.

Parameter	Description
Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom. Note For a single monitored object, you can create one alert configuration for each of the following scopes: all rules, all strong rules, and all weak rules. New rules automatically match the corresponding alert based on their strength. To change one of these alert configurations, you can edit the existing configuration. For a custom scope, you can select up to 200 of the configured rules for the current monitored object.
Alert Configuration Name	The alert configuration name must be unique for a single monitored object and can be up to 256 characters long.
Alert Recipient	Configure alert recipients and notification methods. You must select at least one recipient and one notification method. Alert Recipient: You can select Custom, on-call schedule, or Quality Owner as the alert recipient. You can configure up to five custom alert recipients and up to three on-call schedules. Notification Method: You can choose from different methods, such as phone call, email, SMS, DingTalk, Feishu, WeCom, or a custom channel. These methods can be managed by configuring channel settings.

Click OK to complete the alert configuration.

Alert configuration list

After you configure alerts, you can sort, edit, and delete them in the alert configuration list.

Area	Description
① Policy Area	Configure the alert policy for when a quality rule matches multiple alert configurations: The first matched alert configuration takes effect: If you select this policy, only the first alert configuration that the rule matches takes effect. All other configurations are ignored. You can sort the configured alerts. Click Sort Rules. You can drag the icon in front of the alert configuration name to reorder it or use the icons in the Operations column to move it. From left to right, the icons are: Move to Top and Move to Bottom. After you finish reordering, click Finish Sorting. All matched alert configurations take effect: All alert configurations in the current list apply to the quality rules under the current monitored object. For example, if you configure multiple alert settings and select this policy, the system merges alerts by notification method, alert recipient, and alert rule. In specific cases, if alert recipients are the same and the alert recipient type is Custom or Quality Owner, the alert messages are merged based on the merging policy. Note Alert merging is not supported for on-call schedules. If the quality owner is a user group, alert notifications are automatically sent to all members of the group.
② List Area	Displays the name of the alert configuration, its effective scope, and the specific recipients and methods for each notification type. Effective Scope: Click the View icon in the Effective Scope column to view its details. You can only view the object name and rule name for custom alerts. If the rule is deleted, you cannot view the object name, and we recommend that you update the alert configuration.
③ Operations Area	You can edit and delete configured alerts. Edit: Modify a configured alert. If you change the alert recipients or notification methods, inform the relevant personnel to avoid missing business alerts. Delete: After an alert configuration is deleted, it no longer applies to the rules it matches. Proceed with caution.

View quality reports

Click Quality Report to view the Rule validation overview and Rule validation details for the current quality rule.

You can quickly filter validation details by exception result, partition time, or keywords in the rule or object name.
In the Operations column of the rule validation details list, click the icon to view the validation details for the quality rule.
In the Operations column of the rule validation details list, click the icon to view the execution log for the quality rule.

Manage quality rule permissions

Click Permission Management to configure Details Viewability, which specifies which members can view validation record details, quality rule details, and quality reports.
Details Viewability: You can select All Members or Only members with quality management permissions for the current object.
Click OK to save the permission settings.

Run configuration

Note

If a quality rule task times out, the system automatically terminates it and marks its execution as failed. You can view the corresponding execution log on the validation record details page.
If a quality rule task fails and is retried, you can view the execution logs for each retry attempt on the validation record details page.
All quality rules for a monitored object, including new rules and existing ones that have not yet run, use the latest run configuration.

Click the Run Configuration tab, then click the Edit button at the bottom to configure the parameters.

Parameter	Description
Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom.
Run Timeout	If the total run time of a quality rule task (from start to finish, excluding resource and scheduling wait times) exceeds the set threshold, the system automatically terminates the task and marks it as failed. The system default is 1 hour. You can set a custom duration from 0 to 2 hours, with precision to two decimal places.
Automatic retry on failure	For failed quality rule tasks, you can configure whether to automatically retry the task based on the Number of Retries and Retry Interval. This feature is disabled by default. Enable it to configure the settings. Number of Retries: The maximum number of automatic retries for a failed quality rule task. The default is 1, and you can set an integer from 1 to 3. Retry Interval: The time interval between automatic retries. The default is 0 minutes, and you can set an integer from 0 to 10 minutes.

Click OK to complete the run configuration.

Next steps

After you configure the quality rule, you can view the rule on the data source rule list page. For more information, see View the monitored object list.