Set Up Data Quality Rules, Alerts & Connectivity Checks - Dataphin - Alibaba Cloud - Dataphin

Dataphin allows you to monitor the connectivity and table schema changes of your data sources. You can configure quality rules to periodically check for connectivity issues or schema changes and send alerts when exceptions are detected. This helps you monitor the status of your data source connectivity and table schemas in real time. This topic describes how to configure a quality rule for a data source.

Prerequisites

You must add a monitored object before you can configure a quality rule. For more information, see Add a monitored object.

Limitations

You can create quality rules for any data source in Dataphin. Connectivity tests are supported for all data sources, but table schema change monitoring is available only for specific data sources. For more information, see Data sources supported by Dataphin.

Permissions

Super administrators, quality administrators, custom global roles with Quality Rules - Manage permissions, and data source owners can configure schedules, alerts, and runtime configurations for quality rules.
Quality owners and regular users need synchronous read permissions for the data source. To apply for data source permissions, see Apply for data source permissions.
The supported permissions vary depending on the object type. For more information, see Operation permissions for quality rules.

Trial run vs. regular run

The main differences between a trial run and a regular run are their execution method and where their results are displayed. A trial run is a simulated execution of a quality rule to test its correctness and performance. The results of a trial run are not displayed in the quality report. A run is an execution of a quality rule at a specified time. The results of a run are sent to the quality report.

Quality rule configuration

On the Dataphin homepage, in the top navigation bar, choose Governance > Data Quality.
In the left-side navigation pane, click Quality Rules. On the Data Source page, click the name of the target object to open the Quality Rule Details page.
On the Quality Rule Details page, click Create Quality Rule.

In the Create Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic information	Rule name	Enter a name for the quality rule. The name can be up to 256 characters long.
	Rule severity	You can select weak rule or strong rule. Weak rule: If you select weak rule, an alert is triggered when the quality rule check fails, but downstream task nodes are not blocked. Strong rule: If you select strong rule, an alert is triggered when the quality rule check fails. If there are downstream tasks (in a code-check or task-triggered schedule), they are blocked to prevent data pollution. If there are no downstream tasks (for example, in a time-based schedule), only an alert is triggered.
	Description	Enter a description for the quality rule. The description can be up to 128 characters long.
	Rule template	Only Stability is supported, which includes Data source connectivity monitoring and Table schema change monitoring. Data source connectivity monitoring: Monitors for connection failures to a data source configured in Dataphin. Failures can be caused by issues such as network changes or incorrect usernames and passwords, which can cause tasks to fail. Alerts are triggered for any connectivity changes. Table schema change monitoring: Monitors for schema changes in upstream tables, such as renaming, deleting, adding, or removing fields. Such changes can cause downstream tasks to fail. Alerts are triggered for any schema changes.
	Rule type	The rule type is determined by the template and is used for description and filtering.
	Rule configuration
	Select verification table	This parameter is available when you select the table schema change monitoring rule template. Select the data table to verify based on the database or schema. Note If the data source cannot connect to the Dataphin cluster, perform metadata collection before you configure the quality rule. For more information, see Create and manage metadata collection tasks.
Business attribute configuration	Attribute information	The available business attributes are determined by the configuration of the quality rule attributes. For example, if the governing department field is set to an enumerated type (multi-select) with options like Big Data Department, Business Department, and Technology Department, then this attribute appears as a multi-select dropdown menu when you create the quality rule. If the quality owner field is set to custom input with a maximum length of 256 characters, you can enter up to 256 characters for this attribute when you create the quality rule. If the input method for an attribute field is Range, configure it as follows: Range: This is typically used for continuous numerical or date ranges. You can use the symbols >, >=, <, or <=. For more information about attribute configuration, see Create and manage quality rule attributes.
Scheduling attribute configuration	Scheduling method	You can select a pre-configured schedule. If you have not decided on a scheduling method, you can create the quality rule first and configure the schedule later. To create a new schedule, see Create a schedule.

Click OK to complete the rule configuration.

Rule configuration list

In the rule configuration list, you can view, edit, trial run, run, and delete the rules you have configured.

Area	Description
① Filter and search area	Search by object or rule name. Filter by rule type, rule template, rule severity, trial run status, or effective status. Note If a business attribute is configured as searchable and filterable in the quality rule attributes and is enabled, you can search or filter by that attribute.
② List area	Displays information for each rule, including Object type/name, Rule name/ID, Trial run status, Effective status, Rule type, Rule template, Rule severity, Scheduling type, Related knowledge base documents, and Last updater/time. Click the icon next to the refresh button to select which columns to display. Effective status: To avoid blocking production tasks with a faulty rule, we recommend performing a trial run before enabling a rule. Only enable rules that pass the trial run. When enabled, the rule runs automatically according to its schedule. When disabled, the rule does not run automatically, but you can run it manually. Related knowledge base documents: Click View Details to view information from the associated knowledge base, including table name, verification object, rule, and related documents. You can also search, view, edit, and delete knowledge base documents. For more information, see View a knowledge base.
③ Operations area	You can perform the following operations: View, Clone, Edit, Trial run, Run, Schedule configuration, Associate knowledge base document, and Delete. View: View the rule configuration details. Clone: Quickly clone the rule. Edit: After editing a rule, you must perform a new trial run. Trial run: After a trial run, you can click the icon to View trial run log. Run: After a run, you can view the results in the Verification Records. Schedule configuration: Filter by scheduling type or search by schedule name in the dialog box. You can also edit the schedule. Associate knowledge base document: After a rule is associated with a knowledge base, you can view the related knowledge in the quality rule and governance workbench. You can select a knowledge base that is not already associated. To create one, see Create and manage a knowledge base. Delete: Deleting a monitored object deletes all quality rules under it. This action cannot be undone.
④ Batch operations area	You can perform the following operations in batches: Trial run, Run, Configure schedule, Enable, Disable, Modify business attributes, Associate knowledge base document, and Delete. Trial run: Perform a trial run on multiple rules at once. After the trial run, you can click the icon to View trial run log. Run: Run multiple rules at once. After the run, you can view the results in the Verification Records. Configure schedule: Filter by scheduling type or search by schedule name in the dialog box. You can also edit schedules and apply them to multiple quality rules in a batch. Only rules that are editable in the quality rule list can be modified. Enable: After you enable multiple rules, the selected rules will run automatically according to their schedules. Only rules that are editable in the quality rule list can be enabled. Disable: After you disable multiple rules, the selected rules will not run automatically, but you can run them manually. Only rules that are editable in the quality rule list can be disabled. Modify business attributes: You can batch modify business attributes if the corresponding field's value type is single-select or multi-select. If the value type is multi-select, you can append or modify attribute values. If the value type is single-select, you can directly modify the attribute value. Associate knowledge base document: After a rule is associated with a knowledge base, you can view the related knowledge in the quality rule and governance workbench. You can batch associate a knowledge base with multiple monitored objects. To create a knowledge base, see Create and manage a knowledge base. Delete: You can batch delete quality rules. This action cannot be undone, so proceed with caution. You can only delete rules that are editable in the quality rule list.

Create a schedule

Note

When you configure a schedule for a rule, you can quickly choose from the existing schedules for the current table (up to 20 schedules per table).
A single rule can have a maximum of 10 schedules.
Duplicate schedules with identical configurations are automatically removed.

On the Quality Rule Details page, click the Schedule Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

In the Create Schedule dialog box, configure the parameters.

Parameter	Description
Schedule name	Enter a custom name for the schedule. The name can be up to 64 characters long.
Scheduling type	You can select time-based schedule or task-triggered schedule. Time-based schedule: Performs data quality checks at a scheduled time, either once or periodically. This method is suitable for scenarios where data is generated at fixed intervals. Scheduling cycle: Running quality rules consumes computing resources. We recommend that you avoid running multiple quality rules concurrently to prevent interference with production tasks. The available cycle types are Day, Week, Month, Hour, and Minute. If the system time zone (the time zone in User Center) is different from the schedule time zone (the time zone configured in Management Center > System Settings > Basic Settings), the rule runs according to the system time zone. Task-triggered schedule: Executes the quality rule before or after a specified task runs successfully. You can select tasks of the following node types to trigger the rule: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and database SQL nodes. This method is suitable for scenarios with consistent table modification tasks. Note You can only select tasks from the production environment to trigger a rule. If the rule severity is set to strong, a check failure may affect production tasks. Use this feature with caution. Supported engine types are MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, AsiaInfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp TDH 6.x, and Transwarp TDH 9.3.x. Trigger time: Select when the quality check should be triggered. You can choose Trigger After All Tasks Succeed, Trigger After Each Successful Run of a Task, or Trigger Before Each Run of a Task. Triggering task: The following roles can select a task node from a production project to trigger the rule. You can also search by node output name. Project administrators of Prod/Basic projects, users with the Ops system role in Prod projects, users with the Developer system role in Basic projects, and users with a custom project role that includes Project Quality Management - Quality Rule Management permissions can select task nodes from production projects. Users with a custom global role that includes Quality Rules - Manage permissions can select task nodes from all production projects. Note If you set Trigger Time to Trigger After All Tasks Succeed, we recommend that you select triggering tasks that have the same scheduling cycle. This prevents rule execution delays due to different scheduling cycles, which would delay the output of quality check results.
Scheduling conditions	This feature is disabled by default. When enabled, the system checks the scheduling conditions before a rule runs. The rule runs only if the conditions are met; otherwise, the current run is skipped. Business Date/Execution Date: If you select Time-based Schedule (which does not support Execution Date), Code Check-triggered Schedule, or Fixed Task-triggered Schedule as the schedule type, you can select Standard Calendar or Custom Calendar. For more information about how to create a custom calendar, see Create a public calendar. If you select Standard Calendar, the available conditions are Month, Day of Week, and Date. For example: If you select Custom Calendar, the available conditions are Date Type and Tag. For example: Run type: If you set the schedule type to Code check-triggered schedule or Fixed task-triggered schedule, you can configure the run type by selecting periodic run, backfill run, or manual run. For example, see the following figure. Note You must configure at least one rule. To add a rule, click + Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.

Click OK to complete the schedule configuration.

Schedule configuration list

After you create a schedule, you can view, edit, clone, and delete it in the schedule configuration list.

Area	Description
① Filter and search area	Search by schedule name. Filter by time-based schedule or task-triggered schedule.
② List area	Displays the following information for each schedule: Schedule name, Scheduling type, Last updater, and Last update time.
③ Operations area	You can edit, clone, or delete a schedule. Edit: Modify the configured schedule information. Important Editing a schedule updates all rules that use it. Proceed with caution. Clone: Quickly clone the schedule configuration. Delete: You cannot delete a schedule that is referenced by a quality rule.

Configure alerts

You can configure different alert methods for different rules. For example, you can set phone alerts for strong rule exceptions and SMS alerts for weak rule exceptions. If a rule matches multiple alert configurations, you can define an alert strategy.

Note

You can create up to 20 alert configurations for a single monitored object.

On the Quality Rule Details page, click the Alert Configuration tab, and then click Create Alert Configuration to open the Create Alert Configuration dialog box.

In the Create Alert Configuration dialog box, configure the parameters.

Parameter	Description
Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom. Note For a single monitored object, you can create one alert configuration for each of the scopes: All Rules, All Strong Rules, and All Weak Rules. Newly added rules will automatically use the corresponding alert configuration based on their severity. To change one of these alert configurations, you must edit the existing one. The Custom scope allows you to select from all configured rules under the current monitored object, up to a maximum of 200 rules.
Alert configuration name	The alert configuration name must be unique per monitored object and can be up to 256 characters long.
Alert recipient	Configure the alert recipients and alert methods. You must select at least one alert recipient and one alert method. Alert recipient: You can choose Custom, on-duty roster, or quality owner. You can configure up to five custom alert recipients and up to three on-duty rosters. Alert method: You can choose from different notification methods, including Phone, Email, SMS, DingTalk, Feishu, WeCom, and Custom Channel. These notification methods can be managed through Configure Channel Settings.

Click OK to complete the alert configuration.

Alert configuration list

After you configure an alert, you can sort, edit, and delete it in the alert configuration list.

Area	Description
① Sorting area	Configure the alert policy for cases where a quality rule matches multiple alert configurations: The first matched alert configuration takes effect: With this policy, only the first alert configuration that a rule matches is triggered. All subsequent matches are ignored. You can sort the configured alerts to set their priority. Click Sort rules, then drag and drop the alert configurations using the icon, or use the move icons in the Operations column to move an item to the top or bottom. After adjusting the order, click Finish sorting. All matched alert configurations take effect: All alert configurations in the list that match the quality rule will take effect. For example, if you configure multiple alert configurations and select this option, the system will consolidate alerts based on the alert method, recipient, and rule. If the alert recipient is the same but the recipient type is Custom or quality owner, the alert messages will be merged. Note Alerts for on-duty rosters are not merged. If a quality owner is a user group, alert notifications are sent to all members of the group.
② List area	Displays the name, effective scope, specific recipients for each alert type, and the corresponding alert methods. Effective scope: Click the View icon next to a rule in the effective scope to see its details. You can only view the object and rule names for custom alerts. If a rule has been deleted, its object name cannot be viewed, and we recommend that you update the alert configuration.
③ Operations area	You can edit or delete configured alerts. Edit: Modify a configured alert. If you change the recipients or alert methods, inform the relevant personnel to prevent missed business alerts. Delete: After an alert configuration is deleted, it no longer applies to the rules it matched. Proceed with caution.

View a quality report

Click Quality Report to view the Rule verification overview and Rule verification details for the current quality rule.

You can filter the verification details by exception result, partition time, or rule/object name keyword.
In the Operations column of the verification details list, click the icon to view the verification details of the quality rule.
In the Operations column of the rule verification details list, click the icon to view the execution logs for the quality rule.

Permission management

Click Permission management and configure Viewable details to specify which members can view verification record details, quality rule details, and quality reports.
Viewable details: You can select All members or Only members with quality management permissions for the current object.
Click OK to save the permission settings.

Quality rule runtime configuration

Note

If a quality rule task times out, the system terminates it and marks its status as failed. You can view the corresponding execution log on the verification record details page.
If a rule is configured to retry on failure, you can view the log for each retry attempt on the verification record details page.
For a single monitored object, all quality rules (including new rules and existing rules that have not yet run) follow the latest runtime configuration.

Click the Runtime Configuration tab, then click Edit at the bottom to configure the parameters.

Parameter	Description
Scope	You can select All Rules, All Strong Rules, All Weak Rules, or Custom.
Execution timeout	If a task's runtime (excluding resource and scheduling wait times) exceeds the configured threshold, the system automatically terminates the task and marks it as failed. The default is 1 hour. You can set a custom duration from 0 to 2 hours, accurate to two decimal places.
Automatic retry on failure	You can configure failed tasks to be retried automatically by setting the Number of Retries and Retry Interval. This feature is disabled by default. Number of Retries: The maximum number of automatic retries for a failed task. The default is 1, and you can configure an integer from 1 to 3. Retry Interval: The time interval between each automatic retry. The default is 0 minutes, and you can configure an integer from 0 to 10 minutes.

Click OK to save the runtime configuration.

Next steps

After configuring a quality rule, you can view it in the data source rule list. For more information, see View the monitored object list.