Create data source quality rules - Dataphin - Alibaba Cloud Documentation Center

Dataphin lets you monitor the connectivity of data sources and changes in table schemas. Quality rules periodically check data source connectivity or data structure changes. You can also set alerts for abnormalities. This helps you understand the status of data source connectivity and table schemas in real time. This topic describes how to configure quality rules for data sources.

Prerequisites

You must add a monitored object before you can configure quality rules. For more information, see Add a monitored object.

Limits

You can create quality monitoring rules for any data source in Dataphin. All supported data sources can be tested for connectivity. However, only some data sources support quality rules for monitoring table schema changes. For more information, see Data sources supported by Dataphin.

Permission description

Super administrators, quality administrators, custom global roles with Quality Rule Management permissions, and data source owners can configure scheduling and alerts for quality rules.
Quality owners and regular users require additional read-through permissions for data sources. To request data source permissions, see Request data source permissions.
The operation permissions for different objects vary. For more information, see Quality rule operation permissions.

Differences between a trial run and a regular run

A trial run and a regular run differ in their execution method and how results are displayed. A trial run is a simulated execution of a quality rule to test its correctness and operation. The results of a trial run are not displayed in the quality report. A regular run checks the quality rule at a scheduled time. The results of the run are added to the quality report for you to view and analyze.

Configure a quality rule

From the top menu bar on the Dataphin home page, choose Administration > Data Quality.
In the navigation pane on the left, click Quality Rule. On the Datasource page, click the name of the target object to navigate to the Quality Rule Details page.
On the Quality Rule Details page, click the Create Quality Rule button.

In the Create Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic Information	Rule Name	Enter a custom name for the quality rule. The name can be up to 256 characters long.
	Rule Strength	Both Soft Rules and Strong Rules are supported. Weak Rule: If you select Weak Rule, an alert is triggered if the quality rule check fails, but downstream task nodes are not blocked. Strong Rule: If you select Strong Rule, an alert is triggered if the quality rule check fails. If there are downstream tasks, such as code check scheduling or task trigger scheduling, they are blocked to prevent data pollution. If there are no downstream tasks, such as for recurring quality scheduling, only an alert is triggered.
	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
	Rule Template	Only Stability is supported. It includes Data Source Connectivity Monitoring and Table Structure Change Monitoring. Connectivity Monitoring: Monitors and sends alerts for connectivity changes. Data sources configured in Dataphin may fail to connect because of network changes, or incorrect usernames or passwords. This can cause task errors. Table Structure Change: Monitors and sends alerts for table schema changes. Changes in the schema of ancestor tables, such as renaming, deleting, adding, or removing fields, can cause downstream errors.
	Rule Type	The rule type depends on the template. It is the most basic attribute of the template and can be used for descriptions and filtering.
	Rule Configuration
	Select Verification Table	This parameter is available when the rule template is set to Table Structure Change Monitoring. Select the data table to check based on the DB or schema.
Business Attribute Configuration	Attribute Information	The requirements for entering business attributes depend on how the quality rule attributes are configured. For example, the field for the managing department might be a multi-select enumeration with options such as Big Data Department, Business Department, and Technology Department. When you create a quality rule, this attribute is a multi-select drop-down list with these options. The field for the rule owner might be a custom input with a length of 256 characters. When you create a quality rule, you can enter up to 256 characters for this attribute. If the input method for the attribute field is Range Interval, configure it as follows: Range Interval: This is often used when the value is a continuous number or date. You can select the >, >=, <, or <= symbol. For more information about attribute configurations, see Create and manage quality rule attributes.
Scheduling Attribute Configuration	Scheduling Method	Select a configured schedule. If you have not decided on a scheduling method, you can create the quality rule first and configure it later. To create a new schedule, see Create a schedule.

Click OK to save the rule configuration.

Rule configuration list

The rule configuration list displays information about your configured data source rules. From this list, you can view, edit, delete, run, or perform a trial run on a rule.

Area	Description
①Filter and Search Area	Lets you quickly search by object or rule name. Lets you filter by rule type, rule template, rule strength, trial run status, and effective status. Note If a business attribute is configured to be searchable and filterable and is enabled, you can search or filter based on that attribute.
②List Area	Displays the object type/name, rule name/ID, trial run status, effective status, rule type, rule template, rule strength, schedule type, and related knowledge base document information. Click the icon before the refresh icon to select the fields to display in the rule list. Effective Status: Run a trial before you enable a rule. This prevents incorrect rules from blocking online tasks. Enable the status for rules that pass the trial run. After you enable the effective status, the selected rule runs automatically according to the configured schedule. After you disable the effective status, the selected rule does not run automatically, but you can run it manually. Related Knowledge Base Documents: Click View Details to view the knowledge base information associated with the rule. This includes the table name, check object, rule, and related knowledge base document information. You can also search, view, edit, and delete the knowledge base. For more information, see View a knowledge base.
③Operation Area	You can perform view, clone, edit, trial run, run, schedule configuration, associate knowledge base documents, and delete operations. View: View the rule configuration details. Clone: Quickly clone a rule. Edit: After you edit a rule, run a trial again. Trial Run: After a trial run, click the icon to View The Trial Run Log. Run: After a run, view the check results in Quality Record. Schedule Configuration: In the dialog box, filter by schedule type or quickly search for schedules by name. You can also edit schedules. Associate Knowledge Base Documents: After a rule is associated with a knowledge base, you can view the associated knowledge in the quality rule and administration workbench. You can select knowledge bases that are not associated. To create a knowledge base, see Create and manage knowledge bases. Delete: Deleting this quality rule object deletes all quality rules under the object. This operation cannot be undone.
④Batch Operations Area	You can perform batch operations such as trial run, run, configure schedule, enable, disable, modify business attributes, associate knowledge base documents, and delete. Trial Run: Supports batch trial runs of rules. After a trial run, click the icon to View The Trial Run Log. Run: Supports batch runs of rules. After a run, view the check results in Quality Record. Schedule Configuration: In the dialog box, filter by schedule type or quickly search for schedules by name. You can also edit schedules to configure them for quality rules in batch. You can only modify selected rules that are editable on the quality rule list page. Enable: After you enable the effective status in batch, the selected rules run automatically according to the configured schedule. You can only enable selected rules that are editable on the quality rule list page. Disable: After you disable the effective status in batch, the selected rules do not run automatically, but you can run them manually. You can only disable selected rules that are editable on the quality rule list page. Modify Business Attributes: When the value type for the business attribute field is single-select or multi-select, you can modify business attributes in batch. If the value type is multi-select, you can append or modify attribute values. If the value type is single-select, you can directly modify the attribute value. Associate Knowledge Base Documents: After a rule is associated with knowledge, you can view the associated knowledge in the quality rule and administration workbench. You can configure knowledge bases in batch for monitored objects. To create a knowledge base, see Create and manage knowledge bases. Delete: Supports batch deletion of quality rule objects. This operation cannot be undone, so proceed with caution. You can only delete selected rules that are editable on the quality rule list page.

Create a schedule

Note

When you configure a rule schedule, you can use an existing schedule for the current table as a template. Each table can have a maximum of 20 scheduling rules.
A single rule can have a maximum of 10 schedules.
Dataphin automatically deduplicates identical scheduling configurations.

On the Quality Rule Details page, on the Schedule Configuration tab, click Create Schedule to open the Create Schedule dialog box.

In the Create Schedule dialog box, configure the parameters.

Parameter	Description
Schedule Name	Enter a custom schedule name. The name can be up to 64 characters long.
Schedule Type	Supports Recurrency Triggered and Task Triggered. Recurrency Triggered: Performs timed, recurring quality checks on data based on the set schedule time. This is suitable for scenarios where data output time is relatively fixed. Recurrence: Quality rule execution consumes computing resources. Avoid running multiple quality rules concurrently to prevent affecting the normal operation of production tasks. Scheduling cycles include Day, Week, Month, Hour, and Minute. If the system time zone (the time zone in the User Center) is different from the scheduling time zone (the time zone configured in Management Center > System Settings > Basic Settings), rules are executed according to the system time zone. Task Triggered: Executes the configured quality rules after or before the specified task runs successfully. You can select task types from nodes such as Engine SQL, Offline Pipeline, Python, Shell, Virtual, Datax, Spark_jar, Hive_MR, and Database SQL to trigger tasks. This is applicable when table modification tasks are fixed. Note Fixed task triggers can only select production environment tasks. If the rule strength is set to strong rule, a scheduling task check failure may affect online tasks. Proceed with caution as needed. Trigger Timing: Select the trigger timing for quality detection. You can select Trigger After All Nodes Run Successfully, Trigger After Each Node Runs Successfully, and Trigger Before Each Node Runs. Triggering Task: Project administrators or members with the O&M system role can select task nodes in the production project. You can search by node output name or select from recommended or all tasks. Note If the trigger timing is set to trigger after all tasks run successfully, select tasks with the same scheduling cycle. This avoids rule execution delays and delayed quality check results due to different scheduling cycles.
Schedule Condition	Disabled by default. If enabled, the system checks whether the scheduling conditions are met before the quality rule is scheduled. The schedule proceeds only if the conditions are met. If not, the schedule is ignored. Data Timestamp/Executed On: If the scheduling type is Recurrency Triggered (timed scheduling does not support execution date), Code Check Triggered Scheduling, or Task Triggered, you can configure the date. Select Normal Calendar or Custom Calendar. For more information about how to customize a calendar, see Create a public calendar. If you select Normal Calendar, the conditions can be Month, Week, or Date. See the following figure. If you select Custom Calendar, the conditions can be Date Type or Tag. See the following figure. Instance Type: If the scheduling type is Code Check Triggered Scheduling or Task Triggered, you can configure the instance type. Select Recurring Instance, Data Backfill Instance, or One-time Instance. See the following figure. Note Configure at least one rule. To add a rule, click the + Add Rule button. Configure up to 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.

Click OK to complete the schedule configuration.

Schedule configuration list

After a schedule is created, you can view, edit, clone, and delete it from the schedule configuration list.

Area	Description
①Filter and Search Area	Lets you quickly search by schedule name. Lets you filter by Recurrency Triggered and Fixed Task Triggered Scheduling.
②List Area	Displays the Schedule Name, Schedule Type, Last Updated By, and Last Update Time.
③Operation Area	You can perform edit, clone, and delete operations on the schedule. Edit: Modify the configured schedule information. Important All rule configurations that reference this schedule change synchronously. Proceed with caution. Clone: Quickly copy the schedule configuration. Delete: Schedules referenced by rule configurations cannot be deleted.

Set alerts

You can configure different alert methods for different rules. For example, you can configure phone call alerts for exceptions in strong rules and text message alerts for exceptions in soft rules. If a rule triggers multiple alert configurations at the same time, you can set an alert policy.

Note

You can create a maximum of 20 alert configurations for a single monitored object.

On the Quality Rule Details page, click the Alert Configuration tab and then the Create Alert Configuration button to open the Create Alert Configuration dialog box.

In the Create Alert Configuration dialog box, configure the parameters.

Parameter	Description
Coverage Range	Select All Rules, All Strong Rules, All Weak Rules, or Custom. Note For a single monitored object, you can configure one alert for each of the three ranges: all rules, all strong rules, and all weak rules. New rules automatically match the corresponding alert based on their strength. To change an alert configuration, modify the existing one. The custom range can include all rules configured under the current monitored object, up to 200 rules.
Alert Configuration Name	The alert configuration name must be unique for a single monitored object and can be up to 256 characters long.
Alert Recipients	Configure alert recipients and alert methods. Select at least one alert recipient and one alert method. Alert Recipients: Select custom, shift schedule, or quality owner as alert recipients. You can configure up to 5 custom alert recipients and up to 3 shift schedules. Alert Methods: Select receiving methods such as phone, email, text message, DingTalk, Lark, WeCom, and Configure a Custom Channel. These receiving methods can be controlled through Configure channel settings.

Click OK to complete the alert configuration.

Alert configuration list

After the alert configuration is complete, you can sort, edit, and delete items in the alert configuration list.

Ordinal Number	Description
①Sorting Area	Configure the alert policy for when a quality rule matches multiple alert configurations: Only The First Matched Alert Configuration Takes Effect: If you select this alert policy, only the first alert configuration matched by the rule takes effect. Other configurations are ignored. In this case, you can sort the configured alerts. Click Rule Sorting. You can drag the icon in front of the alert configuration name to sort, or use the icons in the operation column to move items. The icons from left to right are: move to top and move to bottom. After adjusting the alert order, click the Sorting Completed button. All Alert Configurations Take Effect: All alerts in the current alert configuration list take effect for quality rules under the current monitored object. For example, if you configure multiple alert configurations and select this option, the system merges alerts by receiving method, recipient, and rule. If the alert recipient is the same person and the alert method is custom or quality owner, alert messages are merged according to the merge policy. Note Shift schedules do not support alert merging.
②List Area	Displays the name of the alert configuration, its effective range, the specific recipients for each alert type, and their corresponding receiving methods. Effective Range: Click the View icon after the effective range rule to view the rule's effective range. Only custom alerts let you view the configured object and rule names. If the rule is deleted, you cannot view the object name. Update the alert configuration.
③Operation Area	You can edit and delete the configured alerts. Edit: Modify the configured alert information. If you modify the alert recipients and methods, notify the relevant personnel promptly to avoid missing business alerts. Delete: After deletion, the rules matched by this alert configuration no longer take effect. Proceed with caution.

View quality reports

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

You can filter check details by abnormal result, partition time, or a keyword in the rule or object name.
In the Operation column of the rule check details list, click the icon to view the check details of the quality rule.
In the Operation column of the rule check details list, click the icon to view the execution log of the quality rule.

Set quality rule permission management

Click Permission Management and configure the Can View Details permission to specify which members can view check record details, quality rule details, and quality reports.
For Can View Details, select All Members or Only Members With Quality Management Permissions For The Current Object.
Click OK to save the permission management settings.

What to do next

After a quality rule is configured, you can view it on the data source rule list page. For more information, see View the monitored object list.