DataWorks:Monitor data quality

Prerequisites

Before you begin, ensure that you have:

• Synchronized and processed data. For more information, see Synchronize data and Process data.

Step 1: Go to the Configure by Table page

1. Go to the Data Quality page. Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Governance > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.

2. Go to the Configure by Table page. In the left-side navigation pane of the Data Quality page, choose Configure Rules > Configure by Table. On the Configure by Table page, find the dwd_log_info_di_emr table:

   • In the Connection section, select E-MapReduce.

   • On the right side of the page, specify additional filter conditions to locate the table.

3. In the search results, click Rule Management in the Actions column for dwd_log_info_di_emr. The Table Quality Details page appears.

Step 2: Configure a monitor

A monitor defines the data range (partition) to check and how the check is triggered. Set the monitor to run each time the scheduling node that writes data to dwd_log_info_di_emr completes in the production environment.

1. On the Monitor tab, click Create Monitor.

2. Configure the following parameters:

   Parameter	Value	Description
   Data Range	dt=$[yyyymmdd-1]	The partition filter. When the monitor runs, it checks the partition matching this expression—in this case, yesterday's partition.
   Monitoring Rule	(leave blank)	Configured separately in Step 3.
   Trigger Method	Triggered by Node Scheduling in Production Environment	The monitor runs immediately after the dwd_log_info_di_emr node completes. Select the dwd_log_info_di_emr node created during data processing.

   For the full list of monitor parameters, see Configure a monitoring rule for a single table.

Step 3: Configure monitoring rules

The dwd_log_info_di_emr table processes data from ods_raw_log_d_emr. To catch empty partitions before downstream nodes run, create a strong rule that verifies the row count is greater than 0.

A strong rule blocks descendant nodes and triggers an alert when the check fails.

To create the rule:

1. In the Monitor Perspective section of the Rule Management tab, select the raw_log_number_of_table_rows_not_0 monitor, then click Create Rule. The Create Rule panel appears.

   

2. On the System Template tab, find Table is not empty and click Use. The Table is not empty rule checks whether the partition contains at least one row. It is suited for this scenario because it directly detects whether the ancestor node wrote any data to the partition—if the row count is 0, the rule fails. On the right side of the panel, set Degree of Importance to Strong Rule.

   With this setting, if the row count in dwd_log_info_di_emr is 0, an alert is triggered and descendant nodes are blocked from running.

   

3. Click Determine.

   For information about other rule parameters, see Configure a monitoring rule for a single table.

Step 4: Perform a test run

Run a test to verify the rule configuration before the monitor goes live. Perform the test run after creating all rules associated with the monitor.

1. Click Test Run. The Test Run dialog box appears.

2. In the Test Run dialog box, set the Scheduling Time parameter and click Test Run.

3. After the test run completes, click View Details to see the result.

   

Step 5: Subscribe to the monitor

Subscribe to alert notifications so you can ensure data security, data stability, and the timeliness of data generation.

1. On the Rule Management tab, click Alert Subscription. In the Alert Subscription dialog box, select a notification method and a recipient, then click Save.

   

2. To view or modify your subscriptions, choose Quality O&M > Monitor in the left-side navigation pane. On the Monitor page, click My Subscriptions.

What's next

After data processing is complete, use DataAnalysis to visualize the data on a dashboard. For more information, see Visualize data.