Create real-time metadata table rules in Dataphin - Dataphin

This topic describes how to create a quality rule for a real-time metadata table.

Prerequisites

You must add monitored objects before you can configure quality rules. For more information, see Add monitored objects.

Permissions

Super administrators, quality administrators, users with a custom global role that grants the Quality Rule - Manage permission, users with a custom project role that grants the Project Quality Management - Quality Rule Management permission for the project that contains the metadata table, and owners of the real-time metadata table can configure scheduling and alerts for quality rules.
Quality owners and regular users must have read-through permissions for real-time metadata tables. To request permissions for a real-time metadata table, see Request, renew, and return table permissions.
The supported operation permissions vary for different objects. For more information, see Operation permissions for quality rules.

Trial runs vs. scheduled runs

Trial runs and scheduled runs differ in their execution method and how results are displayed. A trial run is a one-time, simulated execution of a quality rule to check its correctness and performance. The results of a trial run are not displayed in the quality report. A scheduled run checks a quality rule at a specific time. The results of a scheduled run are sent to the quality report for you to view and analyze.

Metadata table rules

Rule type	Description
Statistical trend monitoring	Checks data values and data change trends.
Real-time multi-pipeline comparison	In business-critical scenarios, you can use real-time dual-pipeline or tri-pipeline quality rules to monitor data. If an anomaly occurs, O&M engineers can promptly switch to or back up the data. Real-time multi-pipeline comparison rules can monitor issues such as data latency and statistical bias.
Real-time and offline data comparison	When real-time and offline data use the same statistical logic, a real-time and offline data comparison rule can detect differences between the datasets. A significant difference may indicate a data quality issue.

Configure a quality rule

On the Dataphin homepage, in the top menu bar, choose Administration > Data Quality.
In the navigation pane on the left, click Quality Rule. On the Real-time Metadata Table page, click the name of the target object. On the resulting Quality Rule Details page, configure the quality rule.
On the Quality Rule Details page, click Create Quality Rule.

In the Create Quality Rule dialog box, configure the parameters.

Parameter		Description
Basic information	Rule Name	Enter a custom name for the quality rule.
	Rule Strength	Select Soft Rule or Strong Rule. If you select Soft Rule, an alert is triggered if the check result is abnormal, but downstream nodes are not blocked. If you select Strong Rule, an alert is triggered if the check result is abnormal. If there are downstream nodes (from code check scheduling or task-triggered scheduling), the downstream nodes are blocked to prevent data pollution from spreading. If there are no downstream nodes (from periodic quality scheduling), only an alert is triggered.
	Description	Enter a custom description for the quality rule. The description can be up to 128 characters long.
	Rule Template	Select a consistency or stability rule template. Consistency: Includes Stream-Batch Comparison and Real-time Pipeline Comparison. Stability: Includes Real-time Statistical Value Check. For more information, see Real-time metadata table template types.
	Rule Type	The rule type depends on the template and is its most basic property. It can be used for descriptions and filtering.
Rule configuration	Rule Configuration	Configure the rule based on the selected Rule Template. For more information, see Offline pipeline comparison parameter configuration, Multi-pipeline comparison parameter configuration.
Check configuration	Rule Check	After a data quality rule runs, the system compares the result with the exception check configuration. If the result meets the exception conditions, the check fails and triggers subsequent actions, such as alerts. The available metrics for exception checks depend on the template and its configuration. You can use multiple AND or OR conditions. Use no more than three conditions in your configuration. For more information, see Description of check configuration.
Business property configuration	Property information	The format for business properties depends on how the quality rule properties are configured. For example, if the property for the managing department is a multi-select enumeration with the values `Big Data Department`, `Business Department`, and `Technical Department`, the UI shows a multi-select drop-down list with these options. If the property for the rule owner is a custom input field with a length of 256 characters, you can enter up to 256 characters for this property when you create the rule. If the property field is a range interval, configure it as follows: Range interval: Use this for continuous numerical or date ranges. You can select one of four symbols: >, >=, <, or <=. For more information about property configuration, see Create and manage quality rule properties.
Scheduling property configuration	Scheduling Method	Select a pre-configured schedule. If you have not decided on a scheduling method, you can configure it after creating the quality rule. To create a new schedule, see Create a schedule.

Click OK.
Click Preview SQL to view the SQL changes and compare the current configuration with the last saved version.
Note
- The SQL preview is unavailable if key information is missing.
- The left pane shows the SQL preview of the last saved configuration. If no configuration was saved, this pane is empty. The right pane shows the SQL preview of the current configuration.

Rule configuration list

The rule configuration list displays information about configured metadata table rules and lets you perform operations such as view, edit, trial run, run, and delete.

Area	Description
① Filter and Search Area	Quickly search by object or rule name. Filter by rule type, rule template, rule strength, trial run status, or active status. Note If a business property for a quality rule is configured to be searchable and filterable and is enabled, you can search or filter by that property.
② List Area	Displays the object type/name, rule name/ID, trial run status, active status, rule type, rule template, rule strength, schedule type, and related knowledge base document information. Click the icon before the refresh button to select the fields to display in the rule list. Active Status: Run a trial run before you activate a rule. Activate the rule only if the trial run is successful. This prevents incorrect rules from blocking online nodes. After you activate a rule, it runs automatically based on the configured schedule. After you deactivate a rule, it does not run automatically but can be run manually. Related Knowledge Base Documents: Click View Details to view the information in the knowledge base document associated with the rule. This includes the table name, check object, rule, and related knowledge base document information. You can also search, view, edit, and delete knowledge base documents. For more information, see View a knowledge base.
③ Operation Area	You can perform view, clone, edit, trial run, run, schedule configuration, associate knowledge base document, and delete operations. View: View the details of the rule configuration. Clone: Quickly clone a rule. Edit: After you edit a rule, you must run a trial run again. Trial Run: After the trial run, click the icon to view the Trial Run Log. Run: After the run, view the check results in Quality Record. Schedule Configuration: In the dialog box, filter by schedule type or search for a schedule by name. You can also edit schedules. Associate Knowledge Base Document: After you associate a rule with a knowledge base document, you can view the associated document in Data Quality and the administration workbench. You can select knowledge base documents that are not yet associated. To create a document, see Create and manage a knowledge base. Delete: Deletes the quality rule object and all quality rules under it. This operation cannot be undone.
④ Batch Operation Area	You can perform batch trial runs, runs, schedule configurations, activations, deactivations, business property modifications, knowledge base document associations, and deletions. Trial Run: Batch run trial runs for rules. After the trial run, click the icon to view the Trial Run Log. Run: Batch run rules. After the run, view the check results in Quality Record. Schedule Configuration: In the dialog box, filter by schedule type or search for a schedule by name. You can also edit schedules to batch configure schedules for quality rules. You can modify only the selected rules that are editable on the quality rule list page. Enable: After you batch activate rules, the selected rules run automatically based on the configured schedule. You can activate only the selected rules that are editable on the quality rule list page. Disable: After you batch deactivate rules, the selected rules do not run automatically but can be run manually. You can deactivate only the selected rules that are editable on the quality rule list page. Modify Business Properties: You can batch modify business properties if their value type is single-select or multi-select. For multi-select properties, you can append or overwrite values. For single-select properties, you can overwrite the value. Associate Knowledge Base Document: After you associate a rule with a knowledge base document, you can view the associated document in Data Quality and the administration workbench. You can batch configure knowledge base documents for monitored objects. To create a document, see Create and manage a knowledge base. Delete: Batch delete quality rule objects. This operation cannot be undone. Proceed with caution. You can delete only the selected rules that are editable on the quality rule list page.

Create a schedule

Note

When you configure a schedule for a rule, you can reuse an existing schedule from the current table. Each table can have a maximum of 20 scheduling rules.
You can configure a maximum of 10 schedules for a single rule.
If schedule configurations are identical, they are automatically deduplicated.

On the Quality Rule Details page, on the Schedule Configuration tab, click Create Schedule to open the Create Schedule dialog box.

In the Create Schedule dialog box, configure the parameters.

Parameter	Description
Schedule Name	Enter a custom name for the schedule.
Schedule Type	Select Timed scheduling, Data update-triggered scheduling, or Fixed task-triggered scheduling. Timed Scheduling: Runs periodic data quality checks based on a set schedule. This is suitable for scenarios where data is generated at a fixed time. Scheduling Cycle: Running quality rules consumes computing resources. Avoid running multiple quality rules at the same time to prevent affecting the normal operation of production nodes. The scheduling cycle can be Day, Week, Month, Hour, or Minute. If the system time zone (in the User Center) is different from the scheduling time zone (in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Data Update-triggered Scheduling: When any code task runs, the system checks if the run updates the specified check scope of the current table. This is suitable for tables with non-fixed modification tasks or tables that require close monitoring, where every change must be monitored. Note Select the partitions updated by the node as the check scope. For non-partitioned tables, the entire table is checked. The system automatically detects all data changes and performs checks to avoid omissions. Fixed Task-triggered Scheduling: Runs the configured quality rule after or before a specified node runs successfully. You can select nodes of the following types to trigger the rule: DPI engine SQL, Offline Pipeline, Python, Shell, Virtual, Datax, Spark_jar, Hive_MR, and Database SQL. This is suitable for scenarios where table modification tasks are fixed. Note You can select only production environment nodes to trigger the schedule. If a strong rule is configured and the scheduled check fails, online nodes may be affected. Proceed with caution as needed. Trigger Time: Select when to run the quality check. You can select Trigger After All Nodes Succeed, Trigger After Each Node Succeeds, or Trigger Before Each Node Runs. Triggering Task: Project administrators or users with the O&M system role can select nodes in the production project. You can search for nodes by their output names or select them from the recommended or all nodes lists. Note If you select Trigger after all nodes succeed, select triggering nodes that have the same scheduling cycle as the rule. This prevents the rule from running with a delay and producing delayed check results.
Schedule Condition	This feature is disabled by default. If you enable it, the system checks whether the scheduling conditions are met before the quality rule is scheduled. The rule is scheduled only if the conditions are met. Otherwise, the schedule is ignored. Data Timestamp/Execution Date: If you select Timed Scheduling (which does not support execution date), Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling as the schedule type, you can configure a date. You can select Normal Calendar or Custom Calendar. For more information about how to customize a calendar, see Create a public calendar. If you select Normal Calendar, you can set conditions for Month, Week, and Date. For example: If you select Custom Calendar, you can set conditions for Date Type and Tag. For example: Instance Type: If you select Data Update-triggered Scheduling or Fixed Task-triggered Scheduling as the schedule type, you can configure the instance type. You can select Recurring Instance, Data Backfill Instance, or One-time Instance. For example: Note Configure at least one rule. To add a rule, click +Add Rule. You can configure a maximum of 10 scheduling conditions. The relationship between scheduling conditions can be AND or OR.

Click OK.

Schedule configuration list

After you create a schedule, you can view, edit, clone, and delete it from the schedule configuration list.

Area	Description
① Filter and Search Area	Quickly search by schedule name. Filter by Timed Scheduling, Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling.
② List Area	Displays the Schedule Name, Schedule Type, Last Updated By, and Last Updated Time.
③ Operation Area	You can edit, clone, and delete the schedule. Edit: Modify the configured schedule information. Important All rule configurations that reference this schedule are changed synchronously. Proceed with caution. Clone: Quickly copy the schedule configuration. Delete: You cannot delete schedules that are referenced by rule configurations.

Alert configuration

You can configure different alert methods for different rules to distinguish between alerts. For example, you can configure phone call alerts for strong rule exceptions and text message alerts for soft rule exceptions. If a rule matches multiple alert configurations, you can set an alert policy to determine which one takes effect.

Note

You can create a maximum of 20 alert configurations for a single monitored object.

On the Quality Rule Details page, on the Alert Configuration tab, click Create Alert Configuration to open the Create Alert Configuration dialog box.

Configure the parameters in the Create Alert Configuration dialog box.

Parameter	Description
Coverage	Select All Rules, All Strong Rules, All Soft Rules, or Custom. Note For a single monitored object, you can configure one alert for each of the three scopes: all rules, all strong rules, and all soft rules. New rules automatically match the corresponding alert based on their strength. To change an alert configuration, modify the existing one. For a custom scope, you can select any of the configured rules under the current monitored object, up to a maximum of 200 rules.
Alert Configuration Name	The alert configuration name must be unique within a single monitored object and can be up to 256 characters long.
Alert Recipient	Configure the alert recipients and alert method. Select at least one alert recipient and one alert method. Alert Recipient: You can select Custom, Shift schedule, or Quality owner. You can configure a maximum of 5 custom alert recipients and a maximum of 3 shift schedules. Alert Method: You can select Phone, Email, Text message, DingTalk, Lark, WeCom, or Custom Channel. This method can be managed in Configure Channel Settings.

Click OK.

Alert configuration list

After you configure alerts, you can sort, edit, and delete them from the alert configuration list.

Ordinal number	Description
① Sorting Area	Configure the alert policy for when a quality rule matches multiple alert configurations: First Matching Alert Configuration Takes Effect: If you select this policy, only the first alert configuration that the rule matches takes effect. Other configurations are ignored. You can sort the configured alerts. Click Sort Rules. You can drag the icon next to the alert configuration name to sort, or use the icons in the Actions column to move it. The icons from left to right are: Move to Top, Move to Bottom. After you finish sorting, click Done. All Matching Alert Configurations Take Effect: All alert configurations in the list take effect for the quality rules under the current monitored object. For example, if you configure multiple alert configurations and select this policy, the system merges alerts by alert method, alert recipient, and alert rule. Specifically, if the alert recipient is the same person and the alert method is Custom or Quality Owner, alert messages are merged based on the merge policy. Note Alert merging is not supported for shift schedules.
② List Area	Displays the name, effective scope, specific recipients for each alert type, and the corresponding alert method. Effective Scope: For custom alerts, you can view the configured object name and rule name. If the rule is deleted, the object name cannot be viewed. Update the alert configuration.
③ Operation Area	You can edit and delete the configured alerts. Edit: Modify the configured alert information. If you modify the alert recipients or alert method, promptly notify the relevant personnel to avoid missing business alerts. Delete: After deletion, the rules matched by this alert configuration will no longer be effective. Proceed with caution.

View a quality report

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

You can quickly filter check details by exception result, partition time, or keyword in the rule or object name.
In the Actions column of the rule check details list, click the icon to view the check details for the quality rule.
In the Actions column of the rule check details list, click the icon to view the execution log for the quality rule.

Configure permission management for quality rules

Click Permission Management to configure the View Details permission. This permission allows members to view check record details, quality rule details, and quality reports.
View Details: You can select All Members or Members with Quality Management Permissions for the Current Object.
Click OK.

What to do next

After you configure the quality rules, you can view them on the real-time metadata table rule list page. For more information, see View the monitored object list.