DataWorks:Monitor data quality

1. Configure a monitor

A monitor checks whether the data quality in a specified timestamp range (partition) of a table meets your expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When the monitor is run, it searches for the data partitions that match the parameter value and checks whether the data quality meets your expectations.

This means that each time the scheduling job for the ods_raw_log_d table runs, a data quality monitor is triggered. The monitor then uses its associated quality rules to check the data in the corresponding data range and determine whether the data conforms to the quality check rules.

Perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the monitor.

   Key parameters:

   Parameter	Configuration example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ods_raw_log_d node that is created during data synchronization.
   Monitoring Rule	Configure this in a later section.

   Note

   For more information about how to configure data quality monitoring, see Configure a monitoring rule for a single table.

In this example, the goal is to monitor whether the table data generated by daily scheduling tasks meets expectations. Because the table generates data for the previous day, the data is considered as expected if the scheduling time is the current day and the calculated result is for the previous day.

2. Configure monitoring rules

The ods_raw_log_d table stores website access logs of users synchronized from OSS. The table is used as a source table in a user persona analysis scenario. To prevent invalid data processing and data quality issues, you need to create and configure a strong rule that monitors whether the number of rows in the table is greater than 0. This rule helps you determine whether synchronization tasks wrote data to the related partitions in the table.

If the number of rows in the related partitions of the ods_raw_log_d table is 0, an alert is triggered, the ods_raw_log_d node fails, and the downstream tasks of the ods_raw_log_d node are blocked.

Perform the following steps:

1. On the Rule Management tab, in the Monitor Perspective section, select an existing quality monitor, such as raw_log_number_of_table_rows_not_0. Then, click Create Rule.

2. On the System Template Rules tab, find the Table Row Count Is Greater Than 0 rule, click +Use, and then set Degree of importance to Strong rules.

   Note

   In this example, the rule is defined as a strong rule. This indicates that if the number of rows in the ods_raw_log_d table is 0, an alert is triggered and the execution of downstream tasks is blocked.

3. Click Determine.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Perform a test run on the monitor

A test run verifies whether the monitoring rules in the monitor are configured correctly. To ensure that the rules are correct and meet your expectations, you can perform a test run after you create them.

1. On the Rule Management tab, under Monitor Perspective, select the quality monitoring rule that you created, such as raw_log_number_of_table_rows_not_0. Then, click Test Run on the right to open the Test Run dialog box.

2. In the Test Run dialog box, select Scheduling Time and click Test Run.

3. After the test run is complete, click View Details to check whether the test passed.

4. Subscribe to the monitor

Data Quality provides a monitoring and alerting feature. You can subscribe to monitors to receive alert notifications about data quality issues. This lets you resolve issues promptly and ensures data security, stability, and timeliness.

1. On the Rule Management tab, in the Monitor Perspective, select the quality monitoring rule, such as raw_log_number_of_table_rows_not_0, and then click Alert Subscription on the right.

2. As prompted, add a Notification Method and Recipient, and then click Save in the Actions column.

3. After you configure the subscription management settings, in the navigation pane on the left, click Quality O&M > Monitor and select My Subscriptions to view and modify the subscribed tasks.

1. Configure a monitor

A monitor checks whether the data quality in a specified timestamp range (partition) of a table meets your expectations.

In this step, you need to set the Data Range parameter of the quality monitor to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether it meets your expectations.

This means that each time the scheduling task for the ods_user_info_d table runs, the data quality monitor is triggered. The associated quality rules are then used to check whether the data in the corresponding range meets the data quality requirements.

Perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the monitor.

   Key parameters:

   Parameter	Configuration example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ods_user_info_d node that is created during data synchronization.
   Monitoring Rule	Do not configure this now. You will configure it in a later section.

   Note

   For more information about how to configure data quality monitoring, see Configure a monitoring rule for a single table.

2. Configure monitoring rules

The ods_user_info_d table stores basic user information synchronized from RDS for MySQL. The table is used as a source table in a user persona analysis scenario. To prevent invalid data processing and data quality issues, you need to create and configure a strong rule that monitors whether the number of rows in the table is greater than 0. This rule helps you determine whether synchronization tasks wrote data to the related partitions in the table.

After the rule takes effect, if the number of rows in the related partitions of the ods_user_info_d table is 0, an alert is triggered, the ods_user_info_d node fails, and the downstream tasks of the ods_user_info_d node are blocked.

Perform the following steps:

1. On the Rule Management tab, in the Monitor Perspective section, select a monitor. In this example, the user_info_quality_control monitor is selected. Then, click Create Rule.

2. On the System Template Rules tab, find the Table Row Count Is Greater Than 0 rule and click +Use. Then, set Degree of importance to Strong rules.

   Note

   In this example, the rule is defined as a strong rule. This indicates that if the number of rows in the ods_user_info_d table is 0, an alert is triggered and the execution of downstream tasks is blocked.

3. On the System Template Rules tab, find the Unique Value Count, Fixed Value rule, click +Use, and then modify the Rule Scope, Monitoring Threshold, and Degree of importance parameters as follows.

   • Rule Scope: uid(STRING)

   • Monitoring Threshold: Normal threshold = 0

   • Degree of importance: soft rule

4. Click Determine.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Other configurations

The procedures to run a quality monitoring test and subscribe to quality monitoring are the same as those described in Configure monitoring rules for the ods_raw_log_d table.

1. Configure the partition filter expression

A monitor checks whether the data quality in a specified timestamp range (partition) of a table meets your expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether the data quality meets your expectations.

This means that each time the scheduling task for the ads_user_info_1d table is run, quality monitoring is triggered and its associated quality rules are used to check whether the data in the corresponding data range meets the data quality check rules.

Perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the monitor.

   Key parameters:

   Parameter	Configuration Example
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	Triggered by production scheduling. Select the ads_user_info_1d node that is created in Data Processing.
   Monitoring Rule	Do not configure this now. This is configured in a later section.

   Note

   For more information about how to configure data quality monitoring, see Configure a monitoring rule for a single table.

2. Configure monitoring rules

The ads_user_info_1d table is used for user persona analysis. To detect the fluctuation of daily UVs, you need to create and configure a rule that monitors the fluctuation of the number of rows in the aggregate data and a rule that monitors whether the primary key values are unique. This helps you observe the fluctuation of daily UVs and learn about online traffic fluctuations at the earliest opportunity.

When this rule is active, it triggers an alert if duplicate primary keys are detected in the ads_user_info_1d table. The rule also triggers a warning alert if the 7-day row count fluctuation rate is greater than 10% but less than 50%, and a critical alert if the rate exceeds 50%.

Perform the following steps:

1. On the Rule Management tab, in the Monitoring Perspective, select an existing quality monitor, such as ads_user_info_quality_control. Then, click Create Rule.

2. On the System Template Rules tab, find the Number Of Rows, 7-day Volatility rule, click +Use, and then configure the Monitoring Threshold and Degree of importance as follows.

   • Monitoring Threshold:

     • Red Threshold > 50%

     • Orange Threshold > 10%

     • Normal Threshold <= 10%

   • Degree of importance: soft rule

3. On the System Template Rules tab, find the Table Row Count Is Greater Than 0 rule, click +Use, and then set Degree of importance to Strong rules.

4. Click Determine.

   Note

   For more information about configuration items, see Configure rules for a single table.

3. Other configurations

The procedures to perform a test run on a quality monitor and subscribe to a quality monitor are the same as those described in the Configure monitoring rules for the ods_raw_log_d table section.