DataWorks:Configure rules by template

Step 1: Go to the Configure by Template page

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

2. In the left-side navigation pane, choose Rule Setting > Configure by Template to open the Configure by Template page.

   Data Quality provides built-in Table-level and Field Level rule templates. Click View Monitoring Rules for a template to configure monitoring rules for tables or fields in batches.

   The Configure by Template page provides three filters at the top: Keyword Search, Associated Scope, and Quality Dimension. The built-in table-level templates include various timeliness monitoring templates, such as fixed value of table rows, 1-day difference in table rows, 1-day fluctuation rate of table rows, 7-day fluctuation rate of table rows, and 30-day fluctuation rate of table rows. The template description explains the baseline calculation method and comparison logic for each template.

Step 2: Configure monitoring rule properties

1. Select a template to apply to multiple tables or partitions in batches. In the Actions column, click View Monitoring Rules. The Batch Add Monitoring Rules page for that template opens.

2. Configure the Basic Properties of the monitoring rule.

   Parameter	Description
   Connection Type	The data source type for the tables that this rule applies to. Note Templates can be used to configure monitoring rules for the following data sources: MaxCompute, EMR, Hologres, CDH Hive, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, StarRocks, MySQL, Lindorm, SQL Server, and DLF.
   Rule Source	This value is fixed to Built-in Template and cannot be changed. It reflects the rule template you selected. For more information about built-in rule templates, see View built-in rule templates.
   Template
   Rule Name	The system automatically generates a rule name. You can customize the suffix.

3. Configure the advanced properties of the monitoring rule.

   Parameter	Description
   Degree of importance	The severity of the rule. Strong rule: An important rule. By default, a critical anomaly blocks the associated scheduling task. Weak rule: A regular rule. By default, a critical anomaly does not block the associated scheduling task.
   Comparison Method	Defines how the rule validates that the table data meets your expectations. Manual Settings: Customize how the data output is compared with the rule. The available comparison methods vary based on the rule template. The methods displayed on the UI prevail. Supports Numeric Type result comparison, which is typically compared against a fixed value (expected value). Comparison methods include Greater Than, Greater Than or Equal To, Equal To, Unequal To, Less Than, and Less Than or Equal To. You can customize the normal data range (normal threshold) and abnormal data range (red threshold). Supports Fluctuation result comparison, which is typically a range comparison. Comparison methods include Absolute Value, Raise, and Drop. You can customize the normal data range (normal threshold). You can also define an anomaly (orange threshold) and an unexpected result (red threshold) based on the degree of deviation. Intelligent Dynamic Threshold: You do not need to manually configure fluctuation thresholds or expected values. The system uses an intelligent algorithm to automatically determine a reasonable threshold. If a data anomaly is detected, an alert is immediately triggered or the task is blocked. Dynamic thresholds also support strong and weak rules. Note The intelligent dynamic threshold comparison method is supported only for Custom SQL, Custom Scope, and Dynamic Threshold quality rules.
   Monitoring Threshold	If Comparison Method is set to Manual Settings, you can set the Normal threshold and Error Threshold. Normal threshold: If the check result meets the value that you specify, the data check passes. Error Threshold: If the check result meets the value that you specify, the data check fails. If the rule performs a Fluctuation check, you must specify the Warning Threshold. Warning Threshold: If the check result meets the value that you specify, the data is anomalous but does not affect business operations.
   Status	The rule's status, which can be Enable or Deactivate. This setting controls whether the rule runs in the production environment. Important If you set the status to Deactivate, the rule cannot be triggered for a test run or by an associated scheduling task.

4. Click Next to go to the Generate Monitoring Rule page.

Step 3: Add tables or fields

Depending on the table-level rule template or field-level rule template you select, batch add the tables or fields to which you want to apply the rule.

Step 4: Create or associate a quality monitor

A quality monitor defines which rules check a specific data object. The object is a data range within the table to check, such as a specific partition of a partitioned table.

You can configure monitors individually or in batches.

Step 5: Test rule execution

1. Click Generate Monitoring Rule to go to the Verify Monitoring Rule page. On the Verify Monitoring Rule page, you can perform the following operations:

   • Test Run: Verifies that the rule configuration is valid.

     After you create the rules, you can select one or more rules and perform a Test Run. In the Test Run dialog box, select a Scheduled Time (the simulated time at which the check is triggered) and a Resource Group. The system calculates the specific partition values of the table to be checked based on this timestamp and the specified Data Scope. After the configuration is complete, click Test Run to check whether the data in the specified table partition meets the configured data quality rule.

     After a test run, you can click Running Records in the Actions column to view the details of the test run and take appropriate actions.

   • Manage Subscriptions: Defines alert recipients.

     You can send alert notifications by using methods such as Email, Email and SMS, DingTalk Chatbot, DingTalk Chatbot @ALL, Lark Group Chatbot, Enterprise WeChat Chatbot, Custom Webhook, and Telephone.

     Note

     • To add a DingTalk, Lark, or WeCom chatbot, obtain its webhook URL and paste the URL into the alert subscription settings.

     • Only DataWorks Enterprise Edition supports the Custom Webhook method. For information about the message format of Custom Webhook alert notifications that are sent by DataWorks, see Appendix: Webhook message format.

     • When you select Email, Email and SMS, or Telephone as the subscription method, you can specify the Authorized object as Data Quality Monitoring Owner, Shift Schedule, or Scheduling Task Owner.

       • Data Quality Monitoring Owner: Alert notifications are sent to the Quality Monitoring Owner specified in the Basic Configurations section of the current quality monitor.

       • Shift Schedule: When a quality rule check is triggered by an associated node, the system sends an alert notification to the on-duty personnel for the day specified in the on-duty schedule.

       • Scheduling Task Owner: Alert notifications are sent to the Head of the scheduling node associated with the quality monitor.

   • Manage Linked Nodes: Defines the trigger method for the rule.

     You can click Use Recommended Running Mode or Manually Specify Running Mode to associate one or more data quality rules with the scheduling nodes that generate the table data. In Operation Center, these nodes include periodically scheduled instances, manually triggered data backfill instances, and test instances. When a node task runs, the associated rule check is triggered. You can set the rule strength to control whether the node fails and exits, which prevents the spread of dirty data.

     • Recommended Running Mode: The system automatically associates the selected rules with the recommended scheduling nodes based on the data lineage of the nodes that produce the table.

     • Manual Running Mode: You can manually associate the selected rules with specified scheduling nodes.

     Important

     A rule must be associated with a corresponding scheduling node to be triggered automatically.

     In the Associate Scheduling dialog box: select the target Workspace and Task node, click Add, and then click OK to create the association.

   • Delete: You can select and delete one or more rules.

   • View Rule Details: In the Actions column, click View Rule Details to view a rule's details. You can also modify, enable, disable, or delete the rule, set its strength, and view logs.

2. After the test run is successful and a schedule is associated, click Complete Check.