DataWorks:Monitor data quality

1. Configure a quality monitor

A quality monitor checks whether the data in a specified data range (partition) of a target table meets expectations.

In this step, you need to set the Data Range for the monitor to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition and checks whether the data quality meets your expectations.

Each time the scheduling task for the ods_raw_log_d table runs, it triggers the quality monitor. The monitor uses its associated quality rules to check the data in the corresponding data range and determine if it meets the data quality check rules.

Perform the following steps:

1. Click the Quality Monitoring tab, and then click Create Quality Monitor.

2. Configure the quality monitor.

   Key parameters:

   Parameter	Configuration example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ods_raw_log_d node created in Synchronize data.
   Select Quality Rule	Do not configure this now. Configure it in a later section.

   Note

   For more information about how to configure a quality monitor, see Configure rules for a single table.

In this tutorial, the goal is to monitor whether the table data generated by daily scheduling tasks meets expectations. Because the table generates data for the previous day, the data is considered as expected if the scheduled time is the current day and the business time is the previous day.

2. Configure monitoring rules

The ods_raw_log_d table is used to receive website access records synchronized from OSS and serves as a source table for user persona analysis. To prevent invalid processing and quality issues, configure a strong rule for this table to check if the row count is greater than 0. This rule determines whether the sync task has written data to the table's partition.

If the number of rows in the corresponding partition of the ods_raw_log_d table is 0, an alert is triggered, the ods_raw_log_d node fails, and the downstream tasks of the ods_raw_log_d node are blocked.

Perform the following steps:

1. Click the Rule Management tab. In the Quality Monitoring Perspective, select the quality monitor that you created, such as raw_log_number_of_table_rows_not_0. Then, click the Create Rule button on the right to go to the Create Rule page.

2. In System Templates, find the Table Row Count Is Greater Than 0 rule, click Use, and then set Importance to Strong Rule.

   Note

   In this tutorial, the rule is defined as a strong rule. If the row count of the ods_raw_log_d table is 0, an alert is triggered and the execution of downstream tasks is blocked.

3. Click OK.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Test run quality monitoring

A test run verifies whether the check rules in a quality monitor are configured correctly. To ensure that the quality rules are correct and meet expectations, test run the quality monitor after you create the rules.

1. On the Rule Management tab, in the Quality Monitoring Perspective, select the quality monitor that you created, such as raw_log_number_of_table_rows_not_0. Then, click Test Run on the right to open the Test Run dialog box.

2. In the Test Run dialog box, select a Scheduled Time and click Test Run.

3. After the test run is complete, follow the on-screen instructions and click View Details to check whether the test passed.

4. Subscription Quality Monitoring

Data Quality provides a monitoring and alerting feature. Subscribe to quality monitors to promptly receive notifications about quality check exceptions and handle them. This ensures data security, stability, and timely output.

1. On the Rule Management tab, in the Quality Monitoring Perspective, select the quality monitor that you created, such as raw_log_number_of_table_rows_not_0. Then, click Alert Subscription on the right.

2. As prompted on the page, add a Subscription Method and Recipient, and then click Save in the Actions column.

3. After you configure the subscription, in the navigation pane on the left, click O&M > Monitoring. Select My Subscriptions to view and modify your subscribed tasks.

1. Configure a quality monitor

A quality monitor checks whether the data in a specified data range (partition) of a target table meets expectations.

In this step, you need to configure the Data Range of the quality monitor to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether it meets your expectations.

Each time the scheduling task for the ods_user_info_d table runs, it triggers the quality monitor. The monitor uses its associated quality rules to check the data in the corresponding data range and determine if it meets the data quality check rules.

Perform the following steps:

1. On the Quality Monitoring tab, click Create Quality Monitor.

2. Configure the quality monitor.

   Key parameters:

   Parameter	Configuration example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ods_user_info_d node created in Synchronize data.
   Select Quality Rule	Do not configure this now. Configure it in a later section.

   Note

   For more information about how to configure a quality monitor, see Configure rules for a single table.

2. Configure monitoring rules

The ods_user_info_d table is used to receive basic user information synchronized from ApsaraDB RDS for MySQL and serves as a source table for user persona analysis. To prevent invalid processing and quality issues, configure a strong rule for this table to check if the row count is greater than 0. This rule determines whether the sync task has written data to the table's partition.

After the rule takes effect, if the number of rows in the corresponding partitions of the ods_user_info_d table is 0, an alert is triggered, the ods_user_info_d node fails, and the downstream tasks of the ods_user_info_d node are blocked.

Perform the following steps:

1. On the Rule Management tab, in the Quality Monitoring Perspective, select the quality monitor that you created, such as user_info_quality_control. Then, click Create Rule to go to the Create Rule page.

2. In System Template, find the Table Row Count Is Greater Than 0 rule, click Use, and then set Degree Of Importance to Strong Rule.

   Note

   In this tutorial, the rule is defined as a strong rule. If the row count of the ods_user_info_d table is 0, an alert is triggered and the execution of downstream tasks is blocked.

3. In the System Template, find the Unique Value Count, Fixed Value rule, click Use, and then modify the Rule Scope, Monitoring Threshold, and Degree Of Importance as follows.

   • Rule Scope: uid(STRING)

   • Monitoring Threshold: Normal threshold = 0

   • Severity: soft rule

4. Click OK.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Other configurations

The procedures to test run the quality monitor and subscribe to the quality monitor are the same as those described in Configure quality monitoring rules for ods_raw_log_d.

1. Configure a partition filter expression

A quality monitor checks whether the data in a specified data range (partition) of a target table meets expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether the data quality meets your expectations.

Each time the scheduling task for the ads_user_info_1d table runs, it triggers the quality monitor. The monitor uses its associated quality rules to check the data in the corresponding data range and determine if it meets the data quality check rules.

Perform the following steps:

1. On the Quality Monitoring tab, click Create Quality Monitor.

2. Configure the quality monitor.

   Key parameters:

   Parameter	Configuration example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ads_user_info_1d node created in Transform data.
   Select Quality Rule	Do not configure this now. Configure it in a later section.

   Note

   For more information about how to configure a quality monitor, see Configure rules for a single table.

2. Create monitoring rules

The ads_user_info_1d table is used for user persona analysis. To detect daily UV fluctuations, monitor the row count fluctuation of the aggregated data and check the uniqueness of the primary key. This helps you observe daily UV fluctuations and stay informed about online traffic changes.

When the rule is in effect, it triggers an alert if it detects that the primary key in the ads_user_info_1d table is not unique. It triggers a warning alert if the 7-day row count fluctuation rate is greater than 10% and less than 50%. It triggers an error alert if the 7-day row count fluctuation rate is greater than 50%.

Perform the following steps:

1. On the Rule Management tab, in the Quality Monitoring Perspective, select the quality monitor that you created, such as ads_user_info_quality_control. Then, click Create Rule to go to the Create Rule page.

2. In the System Template tab, find the Number Of Rows, 7-day Volatility rule, click +Use, and then configure the Monitoring Threshold and Importance Level as follows.

   • Monitoring Threshold:

     • Error threshold > 50%

     • Warning threshold > 10%

     • Normal threshold <= 10%

   • Importance: soft rule

3. In the System Template, find the Table Row Count Is Greater Than 0 rule, click Use, and then set Importance to Strong Rule.

4. Click OK.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Other configurations

The procedures to test run the quality monitor and subscribe to the quality monitor are the same as those described in Configure quality monitoring rules for ods_raw_log_d.