DataWorks:Configure data quality monitoring

Step 1: Go to rule configuration

1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

2. Go to the rule configuration page for the table.

   In the left-side navigation pane of the Data Quality page, click Configure Rules > Configure by Table. Locate the target table based on the following parameters:

   • Connection: E-MapReduce.

   • Table: In this tutorial, you need to configure monitoring rules for the dwd_log_info_di_emr table.

3. In the search results, find the target table and click Rule Management in the Actions column to open the quality details page for the table.

Step 2: Configure a Monitor

A Monitor checks whether the data in a specified data range (partition) of a target table meets your expectations.

In this step, set the Data Range of the Monitor to dt=$[yyyymmdd-1]. When the Monitor runs, it checks data in the matching partition to determine if it meets the quality requirements.

This means that each time the scheduling node for the dwd_log_info_di_emr table runs, it triggers the Monitor. The Monitor then uses its associated quality rules to validate the data within the specified data range.

Follow these steps:

1. On the Monitor tab, click Create Monitor.

2. Configure the Monitor parameters. The key parameters are described below:

   Parameter	Example
   Data Range	dt=$[yyyymmdd-1]
   Monitoring Rule	You will configure this in a later step.
   Trigger Method	Select Triggered by production scheduling and choose the dwd_log_info_di_emr node that you created in the Process data topic.

   Note

   For more information about how to configure a Monitor, see Configure rules: By table (single table).

Step 3: Configure monitoring rules

The dwd_log_info_di_emr table is used to process data from the ods_raw_log_d_emr table. To prevent invalid processing and quality issues, you configure a Strong Rule based on the Table is not empty template. This rule determines if the ancestor node has written data to the table partition.

When the monitored row count of the corresponding partition in the dwd_log_info_di_emr table is 0, an alert is triggered, and the dwd_log_info_di_emr node fails and exits, blocking the execution of the descendant nodes of the dwd_log_info_di_emr node.

Follow these steps:

1. On the Rule Management tab, under Monitor Perspective, select the Monitor that you created (for example, raw_log_number_of_table_rows_not_0), and then click Create Rule to open the Create Rule panel.

2. On the System Template tab, find the The number of rows is greater than 0. rule and click Use. Then, change the Degree of importance to Strong rules.

   Note

   In this tutorial, the rule is defined as a Strong Rule. This means that if the Monitor finds that the dwd_log_info_di_emr table has zero rows, an alert is triggered, and all descendant nodes are blocked.

3. Click Determine.

   Note

   For more information about other configuration items, see Configure rules: By table (single table).

Step 4: Test the Monitor

You can run a test to verify that the monitoring rules are configured correctly and work as expected.

1. Click Test Run to open the Test Run dialog box.

2. In the Test Run dialog box, select a Data Timestamp and click Test Run.

3. After the test run is complete, follow the on-screen instructions and click View Details to check the results.

   The Current Run Record tab displays information in three sections: Rule Information (which includes the rule name, ID, scope, template, importance, comparison method, and threshold); Detection Task Execution Information (which includes the trigger method, rule trigger time, business date, actual start time, and rule end time); and Quality Collection and Comparison Results (which includes the check status, check result, baseline value, sample value, and check value).

Step 5: Subscribe to the Monitor

Data Quality provides a monitoring and alerting feature. Subscribe to a Monitor to receive timely alerts for quality check failures. This lets you resolve issues quickly and ensure your data is produced securely and on time.

1. On the Rule Management tab, click Subscribe to Alerts. In the Subscribe to Alerts dialog box, select a subscription method and recipient, and then click Save.

2. View and modify subscriptions.

   After you configure the subscription, in the left-side navigation pane, click Quality O&M > Monitor. Then, click the My Subscriptions tab to view and modify your subscriptions.