DataWorks:Configure rules: By table

Step 1: Go to the table quality details page

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

2. Go to the rule configuration page for tables.

   In the navigation pane on the left, click Rule Setting > Configure by Table to go to the rule configuration page.

   1. From the Connection list on the left, select the database that contains the table for which you want to configure rules.

   2. Filter tables by criteria such as database type, database, and table name. Click the target table name, or click Rule Management in the Actions column to open the table's quality details page.

      This page displays all quality monitors and rules for the table. You can filter rules by whether they are associated with a quality monitor and define the run mode for unassociated rules.

Step 2: Create a quality monitor

1. Create a quality monitor.

   You can create a quality monitor in one of the following ways:

   1. Method 1: On the Table Quality Details page, click the Rule Management tab. Click the icon next to Monitor Perspective to create a quality monitor.

   2. Method 2: On the Table Quality Details page, switch to the quality monitoring tab. Click Create Monitor.

2. Configure the parameters for the quality monitor.

   Parameter	Parameter	Description
   Basic Configurations	Monitor Name	Enter a custom name for the quality monitor.
   	Quality Monitoring Owner	Specify an owner for the quality monitor. In an alert subscription, you can set this owner as the recipient for Email, Email and SMS, or Telephone notifications.
   	Monitored Object	The object to be checked for data quality. By default, this is the current table.
   	Data Scope	Use a partition filter expression to define the table partitions to be checked. Non-partitioned table: You do not need to configure this parameter. By default, the Full Table is checked. Partitioned table: The expression must be in the partition_name=partition_value format. The partition value can be a fixed value or a value from Appendix 2: Built-in partition filter expressions. Note This setting is ignored when you configure rules by using a custom template or custom SQL, as the custom SQL itself determines which partitions are checked.
   Select Monitoring Rule	Select Monitoring Rule	Select the data quality rules to apply to the specified data range. Note You can create multiple quality monitors for different partitions and associate each with different data quality rules. If you have not created a data quality rule, you can skip this step. You can create the quality monitor first and then add rules to it later. For more information about how to create a data quality rule, see Step 3. Configure data quality rules.
   Running Settings	Trigger Method	The method that triggers the quality monitor. Triggered by Node Scheduling in Production Environment: Associates the quality monitor with a specified periodically scheduled task in DataWorks Operation Center. After the task runs successfully, the data quality rules in this quality monitor are automatically triggered. Dry-run tasks do not trigger data quality rule checks. Triggered Manually: Manually triggers the data quality rules that are associated with the current quality monitor. Important If the table that you monitor is not a MaxCompute table and you set the Trigger Method to Triggered by Node Scheduling in Production Environment, the selected periodically scheduled task cannot use the public scheduling resource group. Otherwise, an error occurs when the quality monitor runs.
   	Associated Auto Triggered Node	If you set the Trigger Method to Triggered by Node Scheduling in Production Environment, you can configure this parameter to specify an associated scheduling node. After the specified scheduling node runs successfully, the data quality rules are automatically triggered.
   	Resources	The computing resources used to run the data quality rule checks. By default, the data source of the monitored table in the workspace is selected. If you select another data source, make sure that its resources can access the table.
   Handling Policies	Quality Issue Handling Policies	The policy to apply when a data quality issue is detected. Alert: When a data quality issue is detected, an alert notification is sent to the quality monitor's subscribers. The default conditions are: Strong rule - Critical anomaly, Strong rule - Warning anomaly, Strong rule - Check failed, Weak rule - Critical anomaly, Weak rule - Warning anomaly, and Weak rule - Check failed. Blocks: When a data quality issue is detected, the system fails the triggering production scheduling node and blocks its downstream nodes. This action blocks the production pipeline to prevent problematic data from spreading. The default condition is Strong rule - Critical anomaly. Important If you set the policy to Blocks, an alert is also triggered when a data quality issue is detected.
   	Alert Method Configuration	You can send alert notifications via Email, Email and SMS, DingTalk Chatbot, DingTalk Chatbot @ALL, Lark Group Chatbot, Enterprise WeChat Chatbot, Custom Webhook, or Telephone. Note To use a DingTalk, Lark, or Enterprise WeChat chatbot, add the chatbot to obtain its webhook URL, and then paste the URL into the alert subscription settings. The Custom Webhook method is supported only in DataWorks Enterprise Edition. For information about the message format of an alert notification that DataWorks sends via a Custom Webhook, see Appendix: Webhook message format. If you select Email, Email and SMS, or Telephone as the notification method, you can set Authorized object to Data Quality Monitoring Owner, Shift Schedule, or Scheduling Task Owner. Data Quality Monitoring Owner: Sends alert notifications to the Quality Monitoring Owner specified in the Basic Configurations section of the current quality monitor. Shift Schedule: When an associated scheduling node triggers a quality alert, the system sends a notification to the on-duty user for the current day in the shift schedule. Scheduling Task Owner: Alert notifications will be sent to the Head of the scheduling node associated with the quality monitor.

3. Click Save to create the quality monitor.

Step 3: Configure data quality rules

1. On the Table Quality Details page, click the Rule Management tab, select the quality monitor that you created, and then click Create Rule to go to the rule configuration page.

2. Create a data quality rule.

   Data quality provides the following methods to configure rules. Select the method that best fits your business requirements.

   Method 1: System template

   Data quality provides dozens of built-in rule templates. In the pane on the left, click + Use next to a template to quickly create a quality rule. You can add multiple rules at the same time.

   You can click + System template rule at the top and then modify the Template parameter to change the rule template.

   System rule template parameters

   Parameter	Description
   Rule Name	Enter a custom name for the rule.
   Template	Defines the type of rule check to perform on the table. Data quality provides a wide range of built-in table-level and field-level rule templates. For more information, see View built-in rule templates. Note Average, sum, minimum, and maximum values apply only to numeric fields.
   Rule Scope	The scope to which the rule applies. For a table-level rule, the scope is the current table by default. For a field-level rule, you must select a specific field.
   Comparison Method	Specifies how the rule validates the table data. Manual Settings: Allows you to manually set the comparison method and thresholds. The available comparison methods vary by rule template. The options displayed in the UI are definitive. Supports comparison of Numeric Type results, typically against a fixed value (expected value). The comparison methods include Greater Than, Greater Than or Equal To, Equal To, Unequal To, Less Than, and Less Than or Equal To. You can customize the normal data range (normal threshold) and the abnormal data range (critical threshold). Supports comparison of Fluctuation results, which is typically a range comparison. The comparison methods include Absolute Value, Raise, and Drop. You can customize the normal data range (normal threshold). You can also define a warning range for data with anomalies (warning threshold) and a critical range for data that does not meet expectations (critical threshold) based on the degree of deviation. Intelligent Dynamic Threshold: The system uses an intelligent algorithm to automatically determine a reasonable threshold, so you do not need to configure one manually. If the system detects abnormal data, it immediately triggers an alert or blocks the associated task. Dynamic thresholds also support strong and weak rules. Note Only custom SQL, custom range, and dynamic threshold quality rules support the intelligent dynamic threshold comparison method.
   Monitoring Threshold	If you set Comparison Method to Manual Settings, you can specify a Normal threshold and a Error Threshold. Normal threshold: A condition that the check result must meet for the data to be considered normal. Error Threshold: A condition that, if met by the check result, indicates critically abnormal data and triggers a critical alert. If the rule performs a Fluctuation check, you must also specify a Warning Threshold. Warning Threshold: A condition that, if met by the check result, indicates a non-critical anomaly that does not affect business operations.
   Retain problem data	If this feature is enabled and a rule check fails, the system automatically creates a table to store the resulting problem data. Important You can configure the retain problem data feature for MaxCompute and Hologres tables. Only some data quality rules support the retain problem data feature. This feature is affected by the rule status. If the rule is Deactivate, the system does not retain problem data.
   Status	The status of the rule, which can be Enable or Deactivate. This controls whether the rule runs in the production environment. Important If you set the status to Deactivate, the rule cannot be triggered for a test run or by an associated scheduling task.
   Degree of importance	The severity of the rule. Strong rule: An important rule. By default, a critical anomaly blocks the associated scheduling task. Weak rule: A regular rule. By default, a critical anomaly does not block the associated scheduling task.
   Configuration Source	The source of the rule configuration. In this case, the value is Data Quality.
   Description	An optional description of the rule.

   Method 2: Custom template

   Note

   Before you can create a rule by using a custom template, you must go to Quality Assets > Rule Template Library to create a custom rule template. For more information, see Create and manage custom rule templates.

   When you use a custom template, its basic configurations, such as the FLAG parameter and validation SQL, are automatically populated. You can specify a custom Rule Name and configure monitoring thresholds based on the rule type. For example, a numeric rule requires a normal threshold and a critical threshold, while a fluctuation-type rule also requires a warning threshold.

   Custom rule template parameters

   This section describes only the parameters that are unique to custom rule templates. For information about other parameters, see System rule template parameters.

   Parameter	Description
   FLAG parameter	Defines the SET command to execute before the data quality check SQL runs.
   SQL	The SQL validation logic. The query must return a single numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM ${tableName} WHERE ds=$[yyyymmdd]; Note The ${tableName} variable is dynamically replaced with the name of the monitored table. For more information about how to configure a partition filter expression, see Appendix 2: Built-in partition filter expressions. If you create a quality monitor for the table and configure a rule by using this method, the Data Scope specified for the monitor is ignored. The WHERE clause in this SQL determines which partitions are checked.

   Method 3: Custom SQL

   This method allows you to customize the data quality validation logic for the table.

   Custom SQL parameters

   This section describes only the parameters that are unique to custom SQL. For information about other parameters, see System rule template parameters.

   Parameter	Description
   FLAG parameter	Defines the SET command to execute before the data quality check SQL runs.
   SQL	The SQL validation logic. The query must return a single numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM <table_name> WHERE ds=$[yyyymmdd]; Note In your configuration, you must replace <table_name> with the actual name of the table. This SQL statement determines which table is monitored. For more information about how to configure a partition filter expression, see Appendix 2: Built-in partition filter expressions. If you create a quality monitor for the table and configure a rule by using this method, the Data Scope specified for the monitor is ignored. The WHERE clause in this SQL determines which partitions are checked.

   Method 4: Custom script

   Custom script rules support hour-level and minute-level data validation. For information about how to write a script rule, see Use a system rule template. Example:

   - assertion: change 30 minutes ago for max(id) = 15
     name: 30-minute difference in max value of id field is 15

3. (Optional) Add the configured rule to a quality monitor. For more information about quality monitors, see Step 2. Create a quality monitor.

   Note

   A quality rule can be triggered only after you add it to a quality monitor. You can select an existing quality monitor here, or you can select this quality rule in the Select quality rules step when you configure a quality monitor.

4. Click Confirm.

Step 4: Test the rule

You can test a quality monitor's rules in the following ways.

Step 5: Modify alert subscriptions

You configured an alert subscription in Step 2. Create a quality monitor. When a rule is triggered, the system sends a notification to the specified alert recipients. If you need to change the alert recipients, you can modify the alert subscription in one of the following ways.

Appendix 1: Fluctuation rate and variance formulas

• Formula for fluctuation rate: fluctuation rate = (sample value - baseline value) / baseline value

  • sample value: The value of the current day's sample. For example, for a 1-day fluctuation rate check on the table row count of an SQL task, the sample value is the row count of the current day's partition.

  • baseline value: A reference value from historical samples.

  Note

  • If the rule is an SQL task table row count, 1-day fluctuation rate check, the baseline value is the row count of the previous day's partition.

  • If the rule is an SQL task table row count, 7-day average fluctuation rate check, the baseline value is the average row count from the previous 7 days.

• Formula for variance: (current sample value - average of last N days) / standard deviation

  Note

  You can use variance only for numeric types such as BIGINT and DOUBLE.

Appendix 2: Built-in partition expressions

Assume the following scenario:

• The data timestamp (bizdate) is 20240524

• The scheduling time is 10:30:00