DataWorks:Quality monitoring

1. Select the table to monitor

On the quality monitoring node editor page, click Add Table. In the Add Table dialog box, select the table to monitor. You can use the More filter to locate the table faster.

2. Configure the monitoring data scope

• Non-partitioned table: The entire table is monitored by default. You can skip this step.

• Partitioned table: You must select the partition data to monitor. You can use scheduling parameters. Click Preview to verify that the partition filter expression is resolved correctly.

3. Configure data quality monitoring rules

You can create new rules or import existing ones. Configured rules are enabled by default.

• Create a new rule

  Click Create Rule to create a quality monitoring rule based on a template or custom SQL. The following sections describe these methods.

  Built-in template

  You can follow these steps to quickly create a quality monitoring rule from a rule template.

  Note

  You can also find the required rule template in the built-in template list on the left and click + Use to create it.

  Click the + Built-in Template Rule tab on the right side of the dialog box. In the Built-in Templates panel on the left, expand a template category such as Table Row Count, find the target template, and click + Use. In the form on the right, configure the rule parameters and click OK.

  System rule template parameters

  Parameter	Description
  Rule Name	Enter a custom name for the rule.
  Template	Defines the type of rule check to perform on the table. Data quality provides a wide range of built-in table-level and field-level rule templates. For more information, see View built-in rule templates. Note Average, sum, minimum, and maximum values apply only to numeric fields.
  Rule Scope	The scope to which the rule applies. For a table-level rule, the scope is the current table by default. For a field-level rule, you must select a specific field.
  Comparison Method	Specifies how the rule validates the table data. Manual Settings: Allows you to manually set the comparison method and thresholds. The available comparison methods vary by rule template. The options displayed in the UI are definitive. Supports comparison of Numeric Type results, typically against a fixed value (expected value). The comparison methods include Greater Than, Greater Than or Equal To, Equal To, Unequal To, Less Than, and Less Than or Equal To. You can customize the normal data range (normal threshold) and the abnormal data range (critical threshold). Supports comparison of Fluctuation results, which is typically a range comparison. The comparison methods include Absolute Value, Raise, and Drop. You can customize the normal data range (normal threshold). You can also define a warning range for data with anomalies (warning threshold) and a critical range for data that does not meet expectations (critical threshold) based on the degree of deviation. Intelligent Dynamic Threshold: The system uses an intelligent algorithm to automatically determine a reasonable threshold, so you do not need to configure one manually. If the system detects abnormal data, it immediately triggers an alert or blocks the associated task. Dynamic thresholds also support strong and weak rules. Note Only custom SQL, custom range, and dynamic threshold quality rules support the intelligent dynamic threshold comparison method.
  Monitoring Threshold	If you set Comparison Method to Manual Settings, you can specify a Normal threshold and a Error Threshold. Normal threshold: A condition that the check result must meet for the data to be considered normal. Error Threshold: A condition that, if met by the check result, indicates critically abnormal data and triggers a critical alert. If the rule performs a Fluctuation check, you must also specify a Warning Threshold. Warning Threshold: A condition that, if met by the check result, indicates a non-critical anomaly that does not affect business operations.
  Retain problem data	If this feature is enabled and a rule check fails, the system automatically creates a table to store the resulting problem data. Important You can configure the retain problem data feature for MaxCompute and Hologres tables. Only some data quality rules support the retain problem data feature. This feature is affected by the rule status. If the rule is Deactivate, the system does not retain problem data.
  Status	The status of the rule, which can be Enable or Deactivate. This controls whether the rule runs in the production environment. Important If you set the status to Deactivate, the rule cannot be triggered for a test run or by an associated scheduling task.
  Degree of importance	The severity of the rule. Strong rule: An important rule. By default, a critical anomaly blocks the associated scheduling task. Weak rule: A regular rule. By default, a critical anomaly does not block the associated scheduling task.
  Configuration Source	The source of the rule configuration. In this case, the value is Data Quality.
  Description	An optional description of the rule.

  Custom template

  Before you use this method, you must first create a custom rule template in Data Quality > Quality Assets > Rule Template Library. You can then create quality monitoring rules from that template. For more information, see Create and manage custom rule templates.

  The following steps show how to create a data quality rule from a custom template.

  Note

  You can also find the required rule template in the custom template list on the left and click + Use to create it.

  In the Create Rule dialog box, click the + Custom Template Rule tab. Configure Rule Name, Rule Template, Quality Dimension, FLAG Parameter, SQL, Comparison Method, and Monitoring Threshold (including normal and red thresholds), and then click OK.

  Custom rule template parameters

  This section describes only the parameters that are unique to custom rule templates. For information about other parameters, see System rule template parameters.

  Parameter	Description
  FLAG parameter	Defines the SET command to execute before the data quality check SQL runs.
  SQL	The SQL validation logic. The query must return a single numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM ${tableName} WHERE ds=$[yyyymmdd]; Note The ${tableName} variable is dynamically replaced with the name of the monitored table. For more information about how to configure a partition filter expression, see Appendix 2: Built-in partition filter expressions. If you create a quality monitor for the table and configure a rule by using this method, the Data Scope specified for the monitor is ignored. The WHERE clause in this SQL determines which partitions are checked.

  Custom SQL

  This method allows you to define custom data quality validation logic for a table.

  Click the + Custom SQL tab at the top. In the form on the right, configure the Rule Name, Rule Template (select Custom SQL), Quality Dimension, FLAG Parameter, SQL, Comparison Method (choose Manual Settings or Intelligent Dynamic Threshold), and Monitoring Threshold, and then click OK.

  Custom SQL parameters

  This section describes only the parameters that are unique to custom SQL. For information about other parameters, see System rule template parameters.

  Parameter	Description
  FLAG parameter	Defines the SET command to execute before the data quality check SQL runs.
  SQL	The SQL validation logic. The query must return a single numeric value. In the custom SQL, use square brackets to match the partition filter expression of the table. Example: SELECT count(*) FROM <table_name> WHERE ds=$[yyyymmdd]; Note In your configuration, you must replace <table_name> with the actual name of the table. This SQL statement determines which table is monitored. For more information about how to configure a partition filter expression, see Appendix 2: Built-in partition filter expressions. If you create a quality monitor for the table and configure a rule by using this method, the Data Scope specified for the monitor is ignored. The WHERE clause in this SQL determines which partitions are checked.

• Import existing rules

  If monitoring rules for the target table already exist in the Data Quality module, you can import them to quickly clone the rules. If no rules exist, go to Data Quality to create them first. For more information, see Configure rules: By table (single table).

  Note

  This method supports importing multiple rules in bulk and allows for configuring monitoring rules at the table field level.

  Click Import Rule. You can search for and select the rules to import by rule ID or name, rule template, or associated scope (entire table or specific fields).

  After you select the rules, click OK to complete the import.

4. Configure runtime resources

Select the runtime resources for the quality rule checks. This selection determines the data source where the quality monitoring task runs. By default, this is the data source where the monitored table is located.

Exception categories

Exception handling policies

You can configure handling policies for exceptions generated by rule checks:

• Do not ignore: If a specific exception category is detected (for example, a strong rule triggers a critical exception), you can configure the system to stop the current node and set its status to Failed.

  Note

  • After the current node fails, downstream nodes will not be executed. This blocks the production pipeline and prevents the spread of problematic data.

  • You can add multiple exception categories to check for.

  • This policy is typically used when an exception has a major impact and needs to block downstream tasks.

• Ignore: Ignore the exception and continue to execute downstream nodes.

Exception notification methods

You can configure how to receive notifications for exceptions (for example, by email). When an exception occurs, the platform sends a notification through the specified method so you can handle the exception promptly.