DataWorks:Monitor data quality

1. Configure a monitor

You can use a monitor to check whether the quality of data in the specified range (partition) of a table meets your expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When triggered, the monitor checks the data in the specified partition to verify it meets the configured quality rules.

In this case, each time the scheduling node that is used to write data to the ods_raw_log_d_odps table is run, the monitor is triggered and the rules that are associated with the monitor are used to check whether the quality of data in the specified range meets your expectations.

You need to perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the parameters of the monitor.

   The following table describes the key parameters.

   Parameter	Description
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	The trigger method. Set this parameter to Triggered by Node Scheduling in Production Environment and select the ods_raw_log_d_odps node that is created during data synchronization.
   Monitoring Rule	You do not need to configure this parameter. The monitoring rules are configured in the Configure monitoring rules section.

   Note

   For more information about how to configure a monitor, see Configure a monitoring rule for a single table.

2. Configure monitoring rules

The ods_raw_log_d_odps table is used to store website access logs of users synchronized from OSS. The table is used as a source table in a user profile analysis scenario. To prevent invalid data processing and data quality issues, you need to create and configure a strong rule that monitors whether the number of rows in the table is greater than 0. This rule helps you determine whether synchronization tasks wrote data to the related partitions in the table.

If the number of rows in the related partitions of the ods_raw_log_d_odps table is 0, an alert is triggered, the ods_raw_log_d_odps node fails and exits, and the descendant nodes of the ods_raw_log_d_odps node are blocked from running.

You need to perform the following steps:

1. In the Monitor Perspective section of the Rule Management tab, select a monitor. In this example, the raw_log_number_of_table_rows_not_0 monitor is selected. Then, click Create Rule on the right side of the tab. The Create Rule panel appears.

2. On the System Template tab of the Create Rule panel, find the Table is not empty rule and click Use. On the right side of the panel, set the Degree of Importance parameter to Strong Rule.

   Note

   In this example, the rule is defined as a strong rule. This indicates that when the number of rows in the ods_raw_log_d_odps table is found to be 0, an alert is triggered and the descendant nodes are blocked from running.

3. Click Determine.

   Note

   For information about other parameters configured for a monitoring rule, see Configure a monitoring rule for a single table.

3. Perform a test run on the monitor

You can perform a test run to verify whether the configurations of the monitoring rules that are associated with the monitor work as expected. To ensure that the configurations of the rules are correct and meet your expectations, perform a test run on the monitor after you create the rules that are associated with the monitor.

1. In the Monitor Perspective section of the Rule Management tab, select a monitor. In this example, the raw_log_number_of_table_rows_not_0 monitor is selected. Then, click Test Run on the right side of the tab. The Test Run dialog box appears.

2. In the Test Run dialog box, configure the Scheduling Time parameter and click Test Run.

3. After the test run is complete, click View Details to check whether the data passes the test.

4. Subscribe to the monitor

Data Quality provides the monitoring and alerting feature. You can subscribe to monitors to receive alert notifications about data quality issues. This way, you can resolve the data quality issues at the earliest opportunity and ensure data security, data stability, and the timeliness of data generation.

1. In the Monitor Perspective section of the Rule Management tab, select a monitor. In this example, the raw_log_number_of_table_rows_not_0 monitor is selected. Then, click Alert Subscription on the right side of the tab.

2. In the Alert Subscription dialog box, configure the Notification Method and Recipient parameters, and click Save in the Actions column.

3. After the subscription configuration is complete, choose Quality O&M > Monitor in the left-side navigation pane. Then, select My Subscriptions on the Monitor page to view and modify the subscribed monitors.

1. Configure a monitor

You can use a monitor to check whether the quality of data in the specified range (partition) of a table meets your expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When the monitor is run, the monitor searches for the data partitions that match the parameter value and checks whether the quality of the data meets your expectations.

In this case, each time the scheduling node that is used to write data to the ods_user_info_d_odps table is run, the monitor is triggered and the rules that are associated with the monitor are used to check whether the quality of data in the specified range meets your expectations.

You need to perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the parameters of the monitor.

   The following table describes the key parameters.

   Parameter	Description
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	The trigger method. Set this parameter to Triggered by Node Scheduling in Production Environment and select the ods_user_info_d_odps node that is created during data synchronization.
   Monitoring Rule	You do not need to configure this parameter. The monitoring rules are configured in the Configure monitoring rules section.

   Note

   For more information about how to configure a monitor, see Configure a monitoring rule for a single table.

2. Configure monitoring rules

The ods_user_info_d_odps table is used to store basic user information synchronized from ApsaraDB RDS for MySQL. The table is used as a source table in a user profile analysis scenario. To prevent invalid data processing and data quality issues, you need to create and configure a strong rule that monitors whether the number of rows in the table is greater than 0. This rule helps you determine whether synchronization tasks wrote data to the related partitions in the table.

After the monitoring rules take effect, if the number of rows in the related partitions of the ods_user_info_d_odps table is 0, an alert is triggered, the ods_user_info_d_odps node fails and exits, and the descendant nodes of the ods_user_info_d_odps node are blocked from running.

You need to perform the following steps:

1. In the Monitor Perspective section of the Rule Management tab, select a monitor. In this example, the user_info_quality_control monitor is selected. Then, click Create Rule on the right side of the tab. The Create Rule panel appears.

2. On the System Template tab of the Create Rule panel, find the Table is not empty rule and click Use. On the right side of the panel, set the Degree of Importance parameter to Strong Rule.

   Note

   In this example, the rule is defined as a strong rule. This indicates that when the number of rows in the ods_user_info_d_odps table is found to be 0, an alert is triggered and the descendant nodes are blocked from running.

3. On the System Template tab of the Create Rule panel, find the Unique value. fixed value rule and click Use. On the right side of the panel, configure the Rule Scope, Monitoring Threshold, and Degree of Importance parameters.

   • Rule Scope: Set it to uid(STRING).

   • Monitoring Threshold: For the Normal threshold parameter, set the comparison operator to = and the value to 0.

   • Degree of Importance: Set it to Weak rules.

4. Click Determine.

   Note

   For information about other parameters configured for a monitoring rule, see Configure a monitoring rule for a single table.

3. Other configurations

The operations to perform a test run on the monitor and subscribe to the monitor are the same as the operations that are described in the Configure monitoring rules for the ods_raw_log_d_odps table section.

1. Configure a partition filter expression

You can use a monitor to check whether the quality of data in the specified range (partition) of a table meets your expectations.

In this step, you must set the Data Range parameter of the monitor to dt=$[yyyymmdd-1]. When the monitor is run, the monitor searches for the data partitions that match the parameter value and checks whether the quality of the data meets your expectations.

In this case, each time the scheduling node that is used to write data to the ads_user_info_1d_odps table is run, the monitor is triggered and the rules that are associated with the monitor are used to check whether the quality of data in the specified range meets your expectations.

You need to perform the following steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the parameters of the monitor.

   The following table describes the key parameters.

   Parameter	Description
   Data Range	dt=$[yyyymmdd-1]
   Trigger Method	The trigger method. Set this parameter to Triggered by Node Scheduling in Production Environment and select the ads_user_info_1d_odps node that is created during data synchronization.
   Monitoring Rule	You do not need to configure this parameter. The monitoring rules are configured in the Configure monitoring rules section.

   Note

   For more information about how to configure a monitor, see Configure a monitoring rule for a single table.

2. Configure monitoring rules

The ads_user_info_1d_odps table is used for user profile analysis. To detect the fluctuation of daily UVs, you need to create and configure a rule that monitors the fluctuation of the number of rows in the aggregate data in the table and a rule that monitors whether the primary key values are unique for the table. This helps you observe the fluctuation of daily UVs and learn the online traffic fluctuation at the earliest opportunity.

After the monitoring rules take effect, if repeated primary keys exist in the ads_user_info_1d_odps table, an alert is triggered. If the fluctuation rate of the number of rows in the ads_user_info_1d_odps table within seven days is greater than 10% and less than 50%, a warning alert is triggered. If the fluctuation rate of the number of rows in the ads_user_info_1d_odps table within seven days is greater than or equal to 50%, a critical alert is triggered.

You need to perform the following steps:

1. In the Monitor Perspective section of the Rule Management tab, select a monitor. In this example, the ads_user_info_quality_control monitor is selected. Then, click Create Rule on the right side of the tab. The Create Rule panel appears.

2. On the System Template tab of the Create Rule panel, find the Number of rows. 7-day volatility rule and click Use. On the right side of the panel, configure the Monitoring Threshold and Degree of Importance parameters.

   • Monitoring Threshold:

     • For the Red Threshold parameter, set the comparison operator to > and the value to 50%.

     • For the Orange threshold parameter, set the comparison operator to > and the value to 10%.

     • For the Normal threshold parameter, set the comparison operator to <= and the value to 10%.

   • Degree of Importance: Set it to Weak rules.

3. On the System Template tab of the Create Rule panel, find the Table is not empty rule and click Use. On the right side of the panel, set the Degree of Importance parameter to Strong Rule.

4. Click Determine.

   Note

   For information about other parameters configured for a monitoring rule, see Configure a monitoring rule for a single table.

3. Other configurations

The operations to perform a test run on the monitor and subscribe to the monitor are the same as the operations that are described in the Configure monitoring rules for the ods_raw_log_d_odps table section.