All Products
Search

This Product

    DataWorks:Monitor data quality

    Document Center

    DataWorks:Monitor data quality

    Last Updated:Mar 26, 2026

    This tutorial walks you through setting up data quality monitoring rules in DataWorks Data Quality for a daily-scheduled data synchronization node. By the end, you will have two monitoring rules on the ods_user_info_d_emr table: one that blocks downstream nodes when no data arrives, and one that alerts you when duplicate business primary keys appear.

    Prerequisites

    Before you begin, ensure that you have:

    • Synchronized the ApsaraDB RDS for MySQL table ods_user_info_d to the E-MapReduce (EMR) Hive table ods_user_info_d_emr via Data Integration

    • Synchronized the user_log.txt file in Object Storage Service (OSS) to the EMR Hive table ods_raw_log_d_emr via Data Integration

    • Processed the collected data into basic user profile data in Data Studio

    Monitoring requirements

    Data Quality monitors source data changes and extract, transform, and load (ETL) dirty data in a user profile analysis pipeline. The following table lists the monitoring requirements for each table in the pipeline.

    Table

    Monitoring requirement

    ods_raw_log_d_emr

    Strong rule: row count > 0 daily

    ods_user_info_d_emr

    Strong rule: row count > 0 daily; weak rule: business primary key is unique daily

    dwd_log_info_di_emr

    No monitoring rule

    dws_user_info_all_di_emr

    No monitoring rule

    ads_user_info_1d_emr

    Rule monitoring daily row count fluctuation (tracks daily unique visitors)

    This tutorial focuses on ods_user_info_d_emr.

    Step 1: Open the Configure by Table page

    1. Log on to the DataWorks console. In the top navigation bar, select the region. In the left-side navigation pane, choose Data Governance > Data Quality. Select the workspace from the drop-down list, then click Go to Data Quality.

    2. In the left-side navigation pane of the Data Quality page, choose Configure Rules > Configure by Table.

    3. On the Configure by Table page, select E-MapReduce in the Connection section, then search for ods_user_info_d_emr on the right side.

    4. In the search results, click Rule Management in the Actions column. The Table Quality Details page opens.

    Step 2: Configure monitoring rules

    Configure a monitor that covers the daily partition of ods_user_info_d_emr, then add two rules to it.

    Select the monitoring scope

    1. On the Monitor tab, click Create Monitor.

    2. Set Data Range to dt=$[yyyymmdd-1].

      This expression targets the partition generated by the daily scheduling node for the current day. Adjust it if your partition naming convention differs.

    Add monitoring rules

    Add the following two rules. For full configuration details, see Configure a monitoring rule for a single table.

    Rule 1: Row count check (strong rule — blocks downstream on failure)

    1. On the Create Monitor page, click Create Rule. The Create Rule panel opens.

    2. On the System Template tab, find Table is not empty and click Use.

    3. Set Degree of Importance to Strong Rule. When this rule fails (row count = 0), Data Quality triggers an alert and the descendant nodes are blocked from running.

    Rule 2: Business primary key uniqueness check (weak rule — alerts only)

    1. On the System Template tab, find Unique value. fixed value and click Use.

    2. Configure the rule parameters:

      Parameter

      Value

      Rule Scope

      uid(STRING)

      Monitoring Threshold — Normal threshold

      Comparison operator: =, value: 0

      Degree of Importance

      Weak rules

    3. Click Determine to save the monitoring rules.

    Set the trigger method

    Set Trigger Method to Triggered by Node Scheduling in Production Environment and select the ods_user_info_d_emr node created during data synchronization.

    Set the failure handling policy

    Choose one or both options based on your requirements:

    Option

    Behavior

    Block downstream nodes

    Stops the pipeline when a strong rule fails

    Send an alert notification

    Notifies designated recipients when any rule fails

    Click Save to save the monitor configuration.

    Step 3: Test the monitor

    Run a test to verify the rules work as expected before production scheduling takes over.

    1. In the Monitor Perspective section of the Rule Management tab, select the monitor you created.

    2. Click Test Run on the right side. The Test Run dialog box opens.

    3. Set Scheduling Time and click Test Run.

    4. After the test run completes, click View Details to check whether the data passes each rule.

    Step 4: Subscribe to alerts

    Configure who receives alert notifications when a rule violation is detected.

    1. In the Monitor Perspective section of the Rule Management tab, select the monitor.

    2. Click Alert Subscription on the right side.

    3. In the Alert Subscription dialog box, configure Notification Method and Recipient, then click Save in the Actions column.

    4. To view or update your subscriptions, choose Quality O&M > Monitor in the left-side navigation pane, then select My Subscriptions on the Monitor page to view and modify the subscribed monitors.

    What's next

    After the pipeline runs, use DataAnalysis to build a visual dashboard from the processed data. See Visualize data on a dashboard.