All Products
Search

This Product

    DataWorks:Monitor data quality

    Document Center

    DataWorks:Monitor data quality

    Last Updated:Mar 27, 2026

    Configure rules to monitor data quality

    Use Data Quality to catch dirty data before it affects downstream nodes in your user profile analysis pipeline.

    Prerequisites

    Before you begin, ensure that you have:

    • Completed Synchronize data and Process data

    • Basic user information from ApsaraDB RDS for MySQL synchronized into the MaxCompute table ods_user_info_d_odps

    • Website access logs from Object Storage Service (OSS) synchronized into the MaxCompute table ods_raw_log_d_odps

    • User profile data processed and stored in DataStudio

    Background

    Data Quality is an end-to-end platform for checking the data quality of heterogeneous data sources, configuring alert notifications, and managing data sources. When offline MaxCompute data changes, Data Quality checks the data and can block scheduling nodes that propagate dirty data downstream. It also maintains check result history for analysis and grading.

    In this tutorial, Data Quality detects source data anomalies and dirty data generated during the ETL process. The following tables require monitoring:

    Table

    Monitoring requirement

    ods_raw_log_d_odps

    Daily non-zero row count check on raw log data to prevent invalid downstream processing

    ods_user_info_d_odps

    Strong rule: daily row count must be greater than 0; Weak rule: business primary key must be unique daily

    dwd_log_info_di_odps

    No separate rule required

    dws_user_info_all_di_odps

    No separate rule required

    ads_user_info_1d_odps

    Daily row count fluctuation monitoring to observe unique visitor (UV) changes

    How monitoring works

    Each monitored table uses two objects:

    • Monitor: defines the data range (partition) to check and when to trigger the check, tied to a scheduling node in the production environment.

    • Monitoring rule: defines what to check (for example, row count, primary key uniqueness, or row count fluctuation) and the action to take when a threshold is exceeded.

    Strong rules and weak rules produce different outcomes when a threshold is exceeded:

    Rule type

    Critical threshold exceeded

    Other exceptions

    Strong rule

    Scheduling node set to Failed; descendant nodes blocked

    Alert notification sent

    Weak rule

    Alert notification sent; descendant nodes not blocked

    Alert notification sent

    Go to the Configure by Table page

    1. Log on to the DataWorks console. In the top navigation bar, select the target region. In the left-side navigation pane, choose Data Governance > Data Quality. Select the target workspace from the drop-down list and click Go to Data Quality.

    2. In the left-side navigation pane, choose Configure Rules > Configure by Table.

    3. On the Configure by Table page, filter for the target tables:

      • In the Connection section, select MaxCompute.

      • In the MaxCompute category, select the current project in the production environment. This example uses workshop2024_01.

      • Use the search filters to find the ods_raw_log_d_odps, ods_user_info_d_odps, and ads_user_info_1d_odps tables.

    4. In the search results, click Create Monitor in the Actions column for the target table. The Table Quality Details page appears.

    Configure monitoring rules for ods_raw_log_d_odps

    The ods_raw_log_d_odps table stores website access logs synchronized from OSS. Configure a strong rule that blocks downstream nodes if the table is empty, ensuring that only valid log data flows to dependent nodes.

    1. Create a monitor

    1. On the Monitor tab, click Create Monitor.

    2. Configure the key parameters:

      For more information about monitor configuration, see Configure a monitoring rule for a single table.

      Parameter

      Value

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by Node Scheduling in Production Environment — select the ods_raw_log_d_odps node created during data synchronization

      Monitoring Rule

      Leave blank. Configure rules in the next step.

    2. Create a monitoring rule

    This rule checks whether the row count in the daily partition is greater than 0. If the row count is 0, an alert is triggered, the ods_raw_log_d_odps scheduling node fails, and all descendant nodes are blocked.

    1. In the Monitor Perspective section of the Rule Management tab, select the raw_log_number_of_table_rows_not_0 monitor. Click Create Rule on the right side.

    2. On the System Template tab, find Table is not empty and click Use. Set Degree of Importance to Strong Rule.

    3. Click Determine.

      For other rule parameters, see Configure a monitoring rule for a single table.

    3. Run a test

    After creating the rule, run a test to confirm the configuration works as expected.

    1. In the Monitor Perspective section of the Rule Management tab, select the raw_log_number_of_table_rows_not_0 monitor. Click Test Run.

    2. In the Test Run dialog box, set Scheduling Time and click Test Run.

    3. After the test completes, click View Details to verify that the data passes the check.

    4. Subscribe to alerts

    Subscribe to receive notifications when Data Quality detects issues in this table.

    1. In the Monitor Perspective section of the Rule Management tab, select the raw_log_number_of_table_rows_not_0 monitor. Click Alert Subscription.

    2. Configure Notification Method and Recipient, then click Save.

    3. To view your subscriptions, choose Quality O&M > Monitor in the left-side navigation pane and select My Subscriptions.

    Configure monitoring rules for ods_user_info_d_odps

    The ods_user_info_d_odps table stores basic user information synchronized from ApsaraDB RDS for MySQL. Configure two rules: a strong rule that blocks downstream nodes if the table is empty, and a weak rule that alerts when primary key values are not unique.

    1. Create a monitor

    1. On the Monitor tab, click Create Monitor.

    2. Configure the key parameters:

      For more information about monitor configuration, see Configure a monitoring rule for a single table.

      Parameter

      Value

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by Node Scheduling in Production Environment — select the ods_user_info_d_odps node created during data synchronization

      Monitoring Rule

      Leave blank. Configure rules in the next step.

    2. Create monitoring rules

    1. In the Monitor Perspective section of the Rule Management tab, select the user_info_quality_control monitor. Click Create Rule.

    2. On the System Template tab, find Table is not empty and click Use. Set Degree of Importance to Strong Rule. If the row count in the daily partition is 0, an alert is triggered and all descendant nodes are blocked.

    3. On the System Template tab, find Unique value. fixed value and click Use. Configure the following parameters:

      Parameter

      Value

      Rule Scope

      uid(STRING)

      Monitoring Threshold (Normal)

      Comparison operator: =, Value: 0

      Degree of Importance

      Weak rules

    4. Click Determine.

      For other rule parameters, see Configure a monitoring rule for a single table.

    3. Run a test and subscribe to alerts

    Follow the same steps as described in Configure monitoring rules for ods_raw_log_d_odps.

    Configure monitoring rules for ads_user_info_1d_odps

    The ads_user_info_1d_odps table is the final result table for user profile analysis. Configure two rules: a weak rule that monitors daily row count fluctuation to detect UV changes, and a strong rule that ensures the table is not empty.

    The rules apply the following thresholds:

    Condition

    Result

    Row count fluctuation within seven days > 50%

    Critical alert triggered

    Row count fluctuation within seven days > 10% and ≤ 50%

    Warning alert triggered

    Row count fluctuation within seven days ≤ 10%

    Normal

    Repeated primary keys detected

    Alert triggered
    The row count fluctuation check is a weak rule: any threshold breach sends an alert but does not block descendant nodes. The Table is not empty check is a strong rule: if the table is empty, the scheduling node fails and descendant nodes are blocked.

    1. Create a monitor

    1. On the Monitor tab, click Create Monitor.

    2. Configure the key parameters:

      For more information about monitor configuration, see Configure a monitoring rule for a single table.

      Parameter

      Value

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by Node Scheduling in Production Environment — select the ads_user_info_1d_odps node created during data synchronization

      Monitoring Rule

      Leave blank. Configure rules in the next step.

    2. Create monitoring rules

    1. In the Monitor Perspective section of the Rule Management tab, select the ads_user_info_quality_control monitor. Click Create Rule.

    2. On the System Template tab, find Number of rows. 7-day volatility and click Use. Configure the following parameters:

      Parameter

      Operator

      Value

      Red Threshold

      >

      50%

      Orange threshold

      >

      10%

      Normal threshold

      <=

      10%

      Degree of Importance

      —

      Weak rules

    3. On the System Template tab, find Table is not empty and click Use. Set Degree of Importance to Strong Rule.

    4. Click Determine.

      For other rule parameters, see Configure a monitoring rule for a single table.

    3. Run a test and subscribe to alerts

    Follow the same steps as described in Configure monitoring rules for ods_raw_log_d_odps.

    What to do next

    After the data is processed, visualize the results in DataAnalysis. For more information, see Visualize data.