DataWorks:Configure rules by template in batches

Supported data sources

Templates support monitoring rules for the following data sources: MaxCompute, E-MapReduce (EMR), Hologres, CDH Hive, AnalyticDB for PostgreSQL, AnalyticDB for MySQL, StarRocks, MySQL, Lindorm, SQL Server, and Data Lake Formation (DLF).

Before you begin: Choose your path

Use this table to determine your configuration approach before you start.

Intelligent Dynamic Threshold is only available for rules based on custom SQL statements, custom ranges, or dynamic threshold.

How it works

Configuring monitoring rules from a template involves three stages:

1. Select a template and configure rule parameters — Choose a Table-Level or Field-Level built-in template and set the comparison method and thresholds that define what "good data" looks like.

2. Add tables or fields in batches — Select all the tables or fields you want the rule to cover.

3. Associate the rule with a quality monitor — A quality monitor defines the data scope (for example, a specific partition of a partitioned table) and links the rule to a scheduling node that triggers checks automatically.

Step 1: Open the Configure by Template page

1. Log on to the DataWorks console. In the top navigation bar, select the target region. In the left-side navigation pane, choose Data Governance > Data Quality. Select the target workspace from the drop-down list and click Go to Data Quality.

2. In the left-side navigation pane, choose Configure Rules > Configure by Template. Data Quality displays built-in Table Level and Field Level rule templates. Click Configure Monitoring Rules for the template you want to use.

   

Step 2: Configure rule properties

On the Batch Add Monitoring Rules page, configure the following settings.

Step 3: Add tables or fields

Based on the template type you selected, add the tables or fields you want to check.

Step 4: Associate a quality monitor

A quality monitor defines the data scope (the Data Range, such as a specific partition) and the trigger settings for the quality check. Associate each table or field with a quality monitor.

Step 5: Test and finalize

Click Generate Monitoring Rule to open the Verify Monitoring Rule page. From this page:

• Test Run: Select one or more rules and click Test Run. In the dialog box, select a Data Timestamp (the simulated trigger time). The system calculates the partition values based on the specified time and Data Range, then checks whether the data meets the configured rule. After the test run completes, click Running Records in the Actions column to view results.

  

• Subscriptions: Configure alert recipients and notification channels. See the alert method options in Step 4.

• Associated Scheduling: Link rules to scheduling nodes. Use Use Recommended Running Mode to let the system automatically associate rules with the recommended nodes based on data lineage, or use Manually Specify Running Mode to associate rules with specific nodes manually.

  Important

  A rule must be associated with a scheduling node to be triggered automatically.

  

• View Rule Details: Click View Rule Details in the Actions column to open a rule's detail page, where you can modify, start, stop, or delete the rule and view logs.

• Delete: Remove one or more selected rules.

After the test run succeeds and scheduling is associated, click Complete Check.

What's next

After the monitor runs, choose Quality O&M in the left-side navigation pane and click Monitor > Running Records to view the quality check status and complete rule check history for the monitored tables.

Appendix: Webhook message format

This section describes the message format and parameters for alert notifications sent using a Custom Webhook.

{
  "detailUrl": "https://dqc-cn-zhangjiakou.data.aliyun.com/?defaultProjectId=3058#/jobDetail?envType=ODPS&projectName=yongxunQA_zhangbei_standard&tableName=sx_up_001&entityId=10878&taskId=16876941111958fa4ce0e0b5746379cd9bc67999d05f8&bizDate=1687536000000&executeTime=1687694111000",
  "datasourceName": "emr_test_01",
  "engineTypeName": "EMR",
  "projectName": "Project name",
  "dqcEntityQuality": {
    "entityName": "tb_auto_test",
    "actualExpression": "ds=20230625",
    "strongRuleAlarmNum": 1,
    "weakRuleAlarmNum": 0
  },
  "ruleChecks": [
    {
      "blockType": 0,
      "warningThreshold": 0.1,
      "property": "id",
      "tableName": "tb_auto_test",
      "comment": "Test a monitoring rule",
      "checkResultStatus": 2,
      "templateName": "Compare the Number of Unique Field Values Against Expectation",
      "checkerName": "fulx",
      "ruleId": 123421,
      "fixedCheck": false,
      "op": "",
      "upperValue": 22200,
      "actualExpression": "ds=20230625",
      "externalId": "123112232",
      "timeCost": "10",
      "trend": "up",
      "externalType": "CWF2",
      "bizDate": 1600704000000,
      "checkResult": 2,
      "matchExpression": "ds=$[yyyymmdd]",
      "checkerType": 0,
      "projectName": "auto_test",
      "beginTime": 1600704000000,
      "dateType": "YMD",
      "criticalThreshold": "0.6",
      "isPrediction": false,
      "ruleName": "Rule name",
      "checkerId": 7,
      "discreteCheck": true,
      "endTime": 1600704000000,
      "MethodName": "max",
      "lowerValue": 2344,
      "entityId": 12142421,
      "whereCondition": "type!='type2'",
      "expectValue": 90,
      "templateId": 5,
      "taskId": "16008552981681a0d6",
      "id": 234241453,
      "open": true,
      "referenceValue": [
        {
          "discreteProperty": "type1",
          "value": 20,
          "bizDate": "1600704000000",
          "singleCheckResult": 2,
          "threshold": 0.2
        }
      ],
      "sampleValue": [
        {
          "discreteProperty": "type2",
          "bizDate": "1600704000000",
          "value": 23
        }
      ]
    }
  ]
}