All Products
Search
Document Center

DataWorks:Configure monitoring rules for a single table

Last Updated:Nov 28, 2023

Data Quality allows you to configure monitoring rules for a single table to monitor whether the quality of table data meets specific conditions. If data quality issues are detected, nodes on which the issues occur are blocked to prevent dirty data from spreading downstream. This ensures that the generated table data meets your business requirements. After you create and configure a monitoring rule, you can test the rule to check whether the rule configurations work as expected. Data Quality also allows you to subscribe to and copy monitoring rules, view operation logs of partitions, and view the previous check results. This topic describes how to configure monitoring rules for a single table and how to manage the rules.

Background information

Data Quality allows you to create monitoring rules based on a template or create custom rules based on custom SQL statement logic.

  • Template rules: You can create monitoring rules based on 43 built-in rule templates. You can also create monitoring rules based on self-maintained rule templates which are formulated based on your commonly used custom rules. The self-maintained rule templates facilitate quick reuse of custom rules.

  • Custom rules: You can create custom monitoring rules if the built-in rule templates cannot meet your requirements for monitoring the quality of data specified by partition filter expressions.

In most cases, a data table contains large amounts of complex data. To prevent full table scanning, you must create template rules or custom rules based on partition filter expressions to monitor the table data in a specified partition. Before you create a monitoring rule, you must create a partition filter expression. After you create a rule, you can manage the rule. For more information, see the Manage rules section in this topic.

Prerequisites

The metadata of a compute engine is collected. Monitoring rules are configured based on the data tables of a compute engine and are used to monitor the data quality of the tables. Therefore, before you create and configure a monitoring rule, you must collect the metadata of the desired compute engine first. For more information, see Metadata collection.

Limits

  • You can only manually configure monitoring rules. Data Quality cannot automatically generate monitoring rules based on data standards.

  • You can configure monitoring rules only for data sources such as MaxCompute, E-MapReduce (EMR), Hologres, AnalyticDB for PostgreSQL, and Cloudera's Distribution including Apache Hadoop (CDH). After you configure a monitoring rule for a compute engine, the rule can be triggered to check the data quality of tables in the compute engine only if the scheduling nodes that generate the table data are run on the exclusive resource group for scheduling that connects to the compute engine. For information about how to configure an exclusive resource group for scheduling, see Create and use an exclusive resource group for scheduling.

  • To perform data quality checks by using a dynamic threshold rule, you must make sure that 21 days of sampling records are available. If the number of days during which sampling records are accumulated is less than 21, exceptions may occur when you use the dynamic threshold rule. If 21 days of sampling records are not available when you configure a dynamic threshold rule, you can use the data backfill feature to supplement 21 days of sampling records to the dynamic threshold rule after you associate the rule with scheduling nodes.

Go to the Rule Configuration-Configure by Table page

  1. Go to the Data Quality page.

    Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.

  2. Go to the Rule Configuration-Configure by Table page.

    In the left-side navigation pane, choose Rule Management > Configure Rule (by Table). The Configure Rule (by Table) page appears.

    1. The data source list on the left of the page displays all data sources that are associated with the current workspace. You can select the data source to which the desired table belongs.

    2. After you select a data source, the list on the right displays the details of all tables in the data source, including the number of monitoring rules that are enabled for a table and the total number of monitoring rules that are configured for a table.

    3. You can enter a keyword in the search box at the top of the page to search for a table. You can also search for a table based on conditions such as Not Configured or Not Enabled.

    4. After you find the desired table, click Configure Monitoring Rule in the Actions column. The rule configuration page of the table appears.

Create a partition filter expression

In most cases, a data table contains large amounts of complex data. To prevent full table scanning, you must create a monitoring rule based on a partition filter expression to monitor the table data in a specified partition. Before you create and configure the monitoring rule, you must perform the steps shown in the following figure to create a partition filter expression on the rule configuration page of the table.

image.png

  • To configure a monitoring rule for a non-partitioned table, you can use NOTAPARTITIONTABLE as a partition filter expression.

  • To configure a monitoring rule for a partitioned table, you can specify a partition filter expression in the data timestamp format, such as dt=$[yyyymmdd].

The following table provides the formats and descriptions of different types of partition filter expressions.

Type

Format

Description

Partition filter expression for a table with only one level of partitions

Partition key=Partition value

The partition value can be a constant or a built-in parameter expression.

Partition filter expression for a table with multiple levels of partitions

Partition key 1=Partition value/Partition key 2=Partition value/Partition key N=Partition value

The partition value can be a constant or a built-in parameter expression. You must enclose a parameter in brackets [], such as $[yyyymmdd-N]. The partition filter expression must include information about all levels of partitions.

DataWorks provides some built-in partition filter expressions in the data timestamp format that are ready for use. If the built-in partition filter expressions cannot meet your business requirements, you can specify custom partition filter expressions. For more information about partition filter expressions in the data timestamp format, see Supported formats of scheduling parameters. The following table describes the built-in partition filter expressions in the data timestamp format.

Note

Partition filter expressions based on which monitoring rules are created do not support braces, such as ${yyyymmdd-1}.

Partition filter expression

Description

dt=$[yyyymmdd]

Queries the scheduling time.

dt=$[yyyymmdd-1]

Queries the data timestamp.

dt=$[yyyymmddhh24miss]

Queries the scheduling time. The value is accurate to the second.

dt=$[yyyymmddhh24miss-1/24]

Queries the time one hour earlier than the scheduling time. The value is accurate to the second.

dt=$[yyyymmdd]000000

Queries the exact hour of the scheduling time.

dt=$[yyyymmdd-7]

Queries the time one week earlier than the current time. The value is accurate to the day.

dt=$[hh24miss-1/24]

Queries the time one hour earlier than the current time.

dt=$[hh24miss-30/24/60]

Queries the time half an hour earlier than the current time.

dt=$[add_months(yyyymmdd,-1)]

Queries the date that is one month earlier than the current date. The value is accurate to the day.

NOTAPARTITIONTABLE

The partition filter expression used for non-partitioned tables.

After you configure the partition filter expression, click Verify. Data Quality uses the current time to calculate data and verify the partition filter expression. The current time is the scheduling time. Then, you can create a template rule or a custom rule based on the partition filter expression.

Note

You can delete the partition filter expressions that you no longer need to use from the list of partition filter expressions. When you delete the partition filter expressions based on which monitoring rules are created, the rules are also deleted. Proceed with caution.

Create a template rule

You can click Add Monitoring Rule or Quick Create to create a template rule.

  • Add Monitoring Rule: allows you to create and configure a rule used for fine-grained data monitoring. You can configure the parameters such as Rule Type, Thresholds, and Comparison Method for a rule based on your business requirements. You can use this method if you want to perform fine-grained data monitoring.

  • Quick Create: allows you to create and configure coarse-grained table- and field-level rules that are commonly used. You can use this method if you want to quickly use the monitoring rule feature or monitor data quality at a coarse granularity.

image.png

Add Monitoring Rule

If you use this method, you can create a monitoring rule based on a built-in rule template or a rule template library.

Built-in rule templates: A total of 43 table- and field-level template rules are provided by the system. The rules cannot be modified.

Rule template library: The library contains self-maintained rule templates that are formulated based on your commonly used custom rules. The self-maintained rule templates facilitate quick reuse of custom rules. You can modify the templates based on your business requirements.

In this example, a built-in rule template is used to create and configure a monitoring rule. The following table describes the parameters that you can configure for the rule.

Parameter

Description

Rule Name

The custom name of the monitoring rule.

Rule Type

The strength of the monitoring rule. The impacts on the descendant nodes of the node that is associated with the current rule vary based on the strength of the rule.

  • Strong: If the critical threshold is exceeded, critical alerts are reported and descendant nodes are blocked. If the warning threshold is exceeded, warning alerts are reported but descendant nodes are not blocked.

  • Soft: If the critical threshold is exceeded, critical alerts are reported but descendant nodes are not blocked. If the warning threshold is exceeded, warning alerts are not reported and descendant nodes are not blocked.

Auto-Generated Threshold

Specifies whether to use dynamic thresholds. You can configure the parameter based on your business requirements. If you set Auto-Generated Threshold to Yes, you do not need to manually configure the fluctuation thresholds or the expected value. The system determines the appropriate thresholds based on intelligent algorithms. If data exceptions are detected, the system triggers alerts or blocks descendant nodes at the earliest opportunity.

Important
  • You can use the dynamic threshold feature only in DataWorks Enterprise Edition or a more advanced edition. For information about how to purchase the DataWorks service and upgrade the DataWorks edition, see Purchase guide.

  • To perform data quality checks by using a dynamic threshold rule, you must make sure that 21 days of sampling records are available. If the number of days during which sampling records are accumulated is less than 21, exceptions may occur when you use the dynamic threshold rule. If 21 days of sampling records are not available when you configure a dynamic threshold rule, you can use the data backfill feature to supplement 21 days of sampling records to the dynamic threshold rule after you associate the rule with scheduling nodes.

Rule Source

The source for the monitoring rule. Valid values: Built-in Template and Rule Templates.

  • Built-in Template: A total of 43 table- and field-level template rules are provided by the system. The rules cannot be modified. For more information, see Built-in monitoring rule templates.

  • Rule Templates: Rule templates that are formulated based on your commonly used custom rules can be maintained in a rule template library. The self-maintained rule templates facilitate quick reuse of custom rules. You can modify the templates based on your business requirements. For more information, see Create, manage, and use rule templates.

Note

You can select Rule Templates only in DataWorks Enterprise Edition or a more advanced edition. For information about how to purchase the DataWorks service and upgrade the DataWorks edition, see Purchase guide.

Field

The fields to be monitored. You can select all fields in a table or a specific field of a numeric type or non-numeric type.

Template

Data Quality provides 43 built-in table- and field-level monitoring rule templates for you to use. For more information, see Built-in monitoring rule templates.

Note

You can configure field-level monitoring rules of the following types only for numeric fields: average value, sum of values, minimum value, and maximum value.

Comparison Method

The comparison method of the monitoring rule. The comparison method varies based on the rule template. Valid values include Absolute Value, Raise, Drop, Greater Than, and Equal To. You can view the supported comparison methods on the Template Rules tab of the Create Rule panel in the DataWorks console.

  • In most cases, a monitoring rule of the numeric type is used to perform a comparison with a fixed value, which is specified by the Expected Value parameter. Therefore, the Greater Than or Equal To comparison method is used for a monitoring rule of the numeric type in most cases. If you use the comparison method such as Greater Than or Equal To, you must configure the Expected Value parameter.

  • In most cases, a monitoring rule of the fluctuation type is used to perform a comparison with a fluctuation range. Therefore, the Absolute Value, Raise, or Drop comparison method is used for a monitoring rule of the fluctuation type in most cases. If you use the comparison method such as Absolute Value, Raise, or Drop, you must configure the Thresholds parameter.

Thresholds

Used to calculate the fluctuation.

You can calculate the fluctuation by using the following formula: Fluctuation = (Sample value - Baseline)/Baseline.

  • Sample value: The sample value for the current day. For example, if you want to check the fluctuation in the number of table rows on an SQL node within a day, the sample value is the number of table rows in partitions on that day.

  • Baseline: The comparison value collected from the previous N days. Examples:

    • If you want to check the fluctuation in the number of table rows on an SQL node within a day, the baseline is the number of table rows in partitions on the previous day.

    • If you want to check the average fluctuation in the number of table rows on an SQL node within seven days, the baseline is the average number of table rows in the last seven days.

You can specify the warning threshold and critical threshold of the fluctuation to monitor data and identify issues of different severities:

  • Scenario 1: If the value that is monitored is less than or equal to the warning threshold, the data is considered to be normal.

  • Scenario 2: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a weak rule, the data is considered to be normal.

  • Scenario 3: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a strong rule, warning alerts are reported but nodes are not blocked.

  • Scenario 4: If the value that is monitored is greater than the critical threshold and the rule is a weak rule, critical alerts are reported but nodes are not blocked.

  • Scenario 5: If the value that is monitored is greater than the critical threshold and the rule is a strong rule, critical alerts are reported and nodes are blocked.

Start-Stop Status

Specifies whether to apply the monitoring rule in the production environment.

Important

If you turn off the switch for the monitoring rule, you cannot test the rule, and the rule cannot be triggered by scheduling nodes that are associated with the rule.

Retain problem data

If the monitoring rule is enabled and a data quality check based on the rule fails, the system automatically creates a table to store the problematic data that is identified during the data quality check.

Important
  • The Retain problem data parameter is available only for MaxCompute tables.

  • The Retain problem data parameter is available only for specific monitoring rules in Data Quality.

  • If you turn off the Start-Stop Status switch for the monitoring rule, problematic data is not stored.

Quick Create

This method allows you to create and configure coarse-grained table- and field-level rules that are commonly used. You can quickly configure a monitoring rule by using this method.

Parameter

Description

Rule Name

The custom name of the monitoring rule.

Field

The fields to be monitored. You can select all fields in a table or a specific field of a numeric type or non-numeric type.

Trigger

The trigger condition of the monitoring rule. Valid values include The number of columns is greater than 0, Table row number dynamic threshold, and The field value already exists. The valid values of the Trigger parameter are different between table-level monitoring rules and field-level monitoring rules. You can view the valid values of the Trigger parameter on the Template Rules tab of the Create Rule panel in the DataWorks console.

Important

You can select Table row number dynamic threshold only in DataWorks Enterprise Edition or a more advanced edition. For information about how to purchase the DataWorks service and upgrade the DataWorks edition, see Purchase guide.

Start-Stop Status

Specifies whether to apply the monitoring rule in the production environment.

Important

If you turn off the switch for the monitoring rule, you cannot test the rule, and the rule cannot be triggered by scheduling nodes that are associated with the rule.

Create a custom rule

If template rules do not meet your business requirements for monitoring the data quality based on a partition filter expression, you can create custom rules based on your business requirements.

You can click Add Monitoring Rule or Quick Create to create a custom rule.

  • Add Monitoring Rule: allows you to create and configure a rule used for fine-grained data monitoring. You can configure the parameters such as Rule Type, Thresholds, and Comparison Method for a rule based on your business requirements. You can use this method if you want to perform fine-grained data monitoring.

  • Quick Create: allows you to create and configure coarse-grained table- and field-level rules that are commonly used. You can use this method if you want to quickly use the monitoring rule feature or monitor data quality at a coarse granularity.

image.png

Add Monitoring Rule

When you create and configure a custom rule by using this method, you can select All Fields in Table, a specific field, or SQL Statement from the Field drop-down list. The parameters that you need to configure for a monitoring rule vary based on the type of the monitoring rule.

  • The following table describes the parameters that you need to configure for the custom rule when you select All Fields in Table or a specific field from the Field drop-down list.

    Parameter

    Description

    Rule Name

    The custom name of the monitoring rule.

    Rule Type

    The strength of the monitoring rule. The impacts on the descendant nodes of the node that is associated with the current rule vary based on the strength of the rule.

    • Strong: If the critical threshold is exceeded, critical alerts are reported and descendant nodes are blocked. If the warning threshold is exceeded, warning alerts are reported but descendant nodes are not blocked.

    • Soft: If the critical threshold is exceeded, critical alerts are reported but descendant nodes are not blocked. If the warning threshold is exceeded, warning alerts are not reported and descendant nodes are not blocked.

    Field

    The fields to be monitored. In this example, select All Fields in Table or a specific field from the Field drop-down list. If you select All Fields in Table or a specific field from the Field drop-down list, you can use the Filter parameter to configure custom filter conditions based on your business requirements.

    Sampling Method

    The sampling method for the monitoring rule. Valid values include count, count/table_count, and sum. The valid values of the Sampling Method parameter are different between table-level monitoring rules and field-level monitoring rules. You can view the valid values of the Sampling Method parameter on the Custom Rules tab of the Create Rule panel in the DataWorks console.

    Note

    The value of count/table_count is the ratio of the number of table rows that you obtain based on filter conditions to the total number of table rows in the current partition.

    Filter

    The filter conditions. For example, if you want to query the partitions of a table based on a specific data timestamp, you can specify pt=$[yyyymmdd-1] as a filter condition.

    Check type

    The threshold type for the monitoring rule. Valid values: Numeric type, Fluctuation, and Auto-Generated Threshold. If you set Check type to Auto-Generated Threshold, you do not need to manually configure the fluctuation thresholds or the expected value. The system determines the appropriate thresholds based on intelligent algorithms. If data exceptions are detected, the system triggers alerts or blocks descendant nodes at the earliest opportunity.

    Note

    You can select Auto-Generated Threshold only in DataWorks Enterprise Edition or a more advanced edition. For information about how to purchase the DataWorks service and upgrade the DataWorks edition, see Purchase guide.

    Verification Method

    The verification method for the monitoring rule. The verification methods that can be selected vary based on the threshold type. You can view the valid values of the Verification Method parameter on the Custom Rules tab of the Create Rule panel in the DataWorks console.

    Comparison Method

    The comparison methods that can be selected vary based on the threshold type. Valid values include Absolute Value, Raise, Drop, Greater Than, and Equal To. You can view the supported comparison methods on the Custom Rules tab of the Create Rule panel in the DataWorks console.

    • In most cases, a monitoring rule of the numeric type is used to perform a comparison with a fixed value, which is specified by the Expected Value parameter. Therefore, the Greater Than or Equal To comparison method is used for a monitoring rule of the numeric type in most cases. If you use the comparison method such as Greater Than or Equal To, you must configure the Expected Value parameter.

    • In most cases, a monitoring rule of the fluctuation type is used to perform a comparison with a fluctuation range. Therefore, the Absolute Value, Raise, or Drop comparison method is used for a monitoring rule of the fluctuation type in most cases. If you use the comparison method such as Absolute Value, Raise, or Drop, you must configure the Thresholds parameter.

    • If you select Auto-Generated Threshold as the threshold type for the monitoring rule, the system provides an appropriate threshold with which you can compare the current value. When you select Auto-Generated Threshold as the threshold type, you must configure the Algorithm reference sample size parameter. The parameter value is 15 by default.

      Note

      The Algorithm reference sample size parameter specifies the minimum time window for dynamic threshold algorithm models to take effect. Less than 10% of data is allowed to be missing within the minimum time window. If the size of sampled data does not meet the requirements, no alerts are reported. The missing data is automatically supplemented by the algorithm models.

    Thresholds

    Used to calculate the fluctuation.

    You can calculate the fluctuation by using the following formula: Fluctuation = (Sample value - Baseline)/Baseline.

    • Sample value: The sample value for the current day. For example, if you want to check the fluctuation in the number of table rows on an SQL node within a day, the sample value is the number of table rows in partitions on that day.

    • Baseline: The comparison value collected from the previous N days. Examples:

      • If you want to check the fluctuation in the number of table rows on an SQL node within a day, the baseline is the number of table rows in partitions on the previous day.

      • If you want to check the average fluctuation in the number of table rows on an SQL node within seven days, the baseline is the average number of table rows in the last seven days.

    You can specify the warning threshold and critical threshold of the fluctuation to monitor data and identify issues of different severities:

    • Scenario 1: If the value that is monitored is less than or equal to the warning threshold, the data is considered to be normal.

    • Scenario 2: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a weak rule, the data is considered to be normal.

    • Scenario 3: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a strong rule, warning alerts are reported but nodes are not blocked.

    • Scenario 4: If the value that is monitored is greater than the critical threshold and the rule is a weak rule, critical alerts are reported but nodes are not blocked.

    • Scenario 5: If the value that is monitored is greater than the critical threshold and the rule is a strong rule, critical alerts are reported and nodes are blocked.

    Start-Stop Status

    Specifies whether to apply the monitoring rule in the production environment.

    Important

    If you turn off the switch for the monitoring rule, you cannot test the rule, and the rule cannot be triggered by scheduling nodes that are associated with the rule.

  • The following table describes the parameters that you need to configure for the custom rule when you select SQL Statement from the Field drop-down list.

    Parameter

    Description

    Rule Name

    The custom name of the monitoring rule.

    Rule Type

    The strength of the monitoring rule. The impacts on the descendant nodes of the node that is associated with the current rule vary based on the strength of the rule.

    • Strong: If the critical threshold is exceeded, critical alerts are reported and descendant nodes are blocked. If the warning threshold is exceeded, warning alerts are reported but descendant nodes are not blocked.

    • Soft: If the critical threshold is exceeded, critical alerts are reported but descendant nodes are not blocked. If the warning threshold is exceeded, warning alerts are not reported and descendant nodes are not blocked.

    Field

    The fields to be monitored. If you select SQL Statement, you can configure the custom SQL logic for the rule. The return value is the value in a row of a column.

    Sampling Method

    The sampling method for the monitoring rule. You can set this parameter only to SQL Statement.

    Set Flag

    The SET statement that is placed before the SQL statement.

    Custom SQL

    The custom SQL statement to be used. You can specify only a custom SQL statement that returns the value in a row of a column.

    In the custom SQL statement, enclose the partition filter expression in brackets []. Sample custom SQL statement:

    select count(*) from table_name where ds=$[yyyymmdd];

    Note
    • In this statement, table_name indicates the name of the table for which you are configuring a monitoring rule. Replace table_name with the actual table name based on your business requirements.

    • Configure a partition filter expression. For more information, see the Create a partition filter expression section in this topic.

    • The partition filter expression that is used in the custom SQL statement instead of the partition filter expression that you configure in the previous step is used by the monitoring rule that you configure.

    Check type

    The threshold type for the monitoring rule. Valid values: Numeric type, Fluctuation, and Auto-Generated Threshold. If you set Check type to Auto-Generated Threshold, you do not need to manually configure the fluctuation thresholds or the expected value. The system determines the appropriate thresholds based on intelligent algorithms. If data exceptions are detected, the system triggers alerts or blocks descendant nodes at the earliest opportunity.

    Note

    You can select Auto-Generated Threshold only in DataWorks Enterprise Edition or a more advanced edition. For information about how to purchase the DataWorks service and upgrade the DataWorks edition, see Purchase guide.

    Verification Method

    The verification method for the monitoring rule. The verification methods that can be selected vary based on the threshold type. You can view the valid values of the Verification Method parameter on the Custom Rules tab of the Create Rule panel in the DataWorks console.

    Comparison Method

    The comparison methods that can be selected vary based on the threshold type. Valid values include Absolute Value, Raise, Drop, Greater Than, and Equal To. You can view the supported comparison methods on the Custom Rules tab of the Create Rule panel in the DataWorks console.

    • In most cases, a monitoring rule of the numeric type is used to perform a comparison with a fixed value, which is specified by the Expected Value parameter. Therefore, the Greater Than or Equal To comparison method is used for a monitoring rule of the numeric type in most cases. If you use the comparison method such as Greater Than or Equal To, you must configure the Expected Value parameter.

    • In most cases, a monitoring rule of the fluctuation type is used to perform a comparison with a fluctuation range. Therefore, the Absolute Value, Raise, or Drop comparison method is used for a monitoring rule of the fluctuation type in most cases. If you use the comparison method such as Absolute Value, Raise, or Drop, you must configure the Thresholds parameter.

    • If you select Auto-Generated Threshold as the threshold type for the monitoring rule, the system provides an appropriate threshold with which you can compare the current value. When you select Auto-Generated Threshold as the threshold type, you must configure the Algorithm reference sample size parameter. The parameter value is 15 by default.

      Note

      The Algorithm reference sample size parameter specifies the minimum time window for dynamic threshold algorithm models to take effect. Less than 10% of data is allowed to be missing within the minimum time window. If the size of sampled data does not meet the requirements, no alerts are reported. The missing data is automatically supplemented by the algorithm models.

    Thresholds

    Used to calculate the fluctuation.

    You can calculate the fluctuation by using the following formula: Fluctuation = (Sample value - Baseline)/Baseline.

    • Sample value: The sample value for the current day. For example, if you want to check the fluctuation in the number of table rows on an SQL node within a day, the sample value is the number of table rows in partitions on that day.

    • Baseline: The comparison value collected from the previous N days. Examples:

      • If you want to check the fluctuation in the number of table rows on an SQL node within a day, the baseline is the number of table rows in partitions on the previous day.

      • If you want to check the average fluctuation in the number of table rows on an SQL node within seven days, the baseline is the average number of table rows in the last seven days.

    You can specify the warning threshold and critical threshold of the fluctuation to monitor data and identify issues of different severities:

    • Scenario 1: If the value that is monitored is less than or equal to the warning threshold, the data is considered to be normal.

    • Scenario 2: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a weak rule, the data is considered to be normal.

    • Scenario 3: If the value that is monitored is greater than the warning threshold and is less than or equal to the critical threshold, and the rule is a strong rule, warning alerts are reported but nodes are not blocked.

    • Scenario 4: If the value that is monitored is greater than the critical threshold and the rule is a weak rule, critical alerts are reported but nodes are not blocked.

    • Scenario 5: If the value that is monitored is greater than the critical threshold and the rule is a strong rule, critical alerts are reported and nodes are blocked.

    Start-Stop Status

    Specifies whether to apply the monitoring rule in the production environment.

    Important

    If you turn off the switch for the monitoring rule, you cannot test the rule, and the rule cannot be triggered by scheduling nodes that are associated with the rule.

Quick Create

This method allows you to create and configure coarse-grained field-level rules that are commonly used. You can quickly configure a monitoring rule by using this method.

Parameter

Description

Rule Name

The custom name of the monitoring rule.

Trigger

The trigger condition for the monitoring rule. You can select only Values Duplicated in Multiple Fields.

Field

The fields to be monitored. You can select multiple fields.

Start-Stop Status

Specifies whether to apply the monitoring rule in the production environment.

Important

If you turn off the switch for the monitoring rule, you cannot test the rule, and the rule cannot be triggered by scheduling nodes that are associated with the rule.

Associate a monitoring rule with scheduling nodes

To monitor the quality of offline data generated by scheduling nodes, you must associate a monitoring rule that is created based on a partition filter expression with the scheduling nodes that generate table data.

  • You can associate a monitoring rule with scheduling nodes that generate table data only after you deploy the scheduling nodes.

  • Before you can associate a monitoring rule with scheduling nodes in a workspace, you must be assigned at least one of the Workspace Administrator, Development, and O&M roles in the workspace.

Data Quality allows you to associate a partition filter expression with nodes. You can associate a partition filter expression with multiple nodes. After you associate a partition filter expression with nodes, the monitoring rules that you created based on the partition filter expression are automatically triggered to check quality of data generated by the nodes when the nodes are run.

Note
  • Data Quality allows you to associate a monitoring rule with nodes in a flexible manner. You can select a node that is not related to your table.

  • If you associate a partition filter expression with multiple nodes, the monitoring rules that you created based on the partition filter expression are triggered when each node finishes running.

image.png

Test the monitoring rule

After you configure the monitoring rule, you can select the scheduling time and test the rule to check whether the partition data in the test run is as expected. If the partition data does not meet your business requirements, adjust the rule.

  1. Go to the rule details page.

    On the Rules page, specify filter conditions to search for the desired rule and click the rule name to go to the rule details page.

  2. Test the monitoring rule.

    On the rule details page, select the rule and click Test. In the Test dialog box, configure the Scheduling Time parameter and click Test. Follow the on-screen instructions to go to the Node Query page to view the test results. The operation details are shown in the following figure.

    Note

    Monitoring rules are configured based on table partitions. If a partition hits multiple monitoring rules, you can select and test the rules at the same time.

    image.png

Manage the monitoring rule

After you create and configure the monitoring rule, you can manage the rule.

Subscribe to the rule

By default, only the user who creates a rule can receive notifications after the rule is triggered. If you want other users to receive notifications, you can click Manage Subscriptions on the rule configuration page to add other users.

Data Quality supports the following notification methods: Email, Email and SMS, DingTalk Chatbot, DingTalk Chatbot @ALL, Lark Group Chatbot, Enterprise WeChat Chatbot, and Custom Webhook.

Note
  • Add a DingTalk chatbot, Lark chatbot, or WeChat chatbot and obtain a webhook URL. Then, copy the webhook URL to the Recipient field in the Manage Subscriptions dialog box.

  • The Custom Webhook notification method is supported only in DataWorks Enterprise Edition. For information about the message format of an alert notification sent by using a custom webhook, see the "Appendix: Message format of alert notifications sent by using a custom webhook URL" section in Configure monitoring rules for multiple tables by template.

View operation logs of partitions

On the rule configuration page of the table, click View Operation Log. In the View Operation Logs panel, you can view the details about historical operations. The Details column displays all monitoring rules that are configured based on the current partition filter expression.

View the previous check results

On the rule configuration page of the table, click View Check Results to go to the Node Query page. On this page, you can view the check results of all monitoring rules that are configured based on the current partition filter expression and historical results.

Clone the monitoring rule

On the rule configuration page of the table, click Clone Rules. In the Clone Rules dialog box, you can configure the New Expression parameter to determine the partition filter expression to which you want to copy the rule. You can also specify whether to synchronize the subscriber of the rule and replace the table name in the rule for which you select SQL Statement for the Field parameter.

Note

This operation copies all the monitoring rules that you created based on the current partition filter expression to the selected partition filter expression.

What to do next

If you do not want the table data that fails the check of a monitoring rule to affect node running during a specified period of time, you can use the noise reduction management feature. For more information, see Manage noise reduction rules.