All Products
Search

This Product

    DataWorks:Monitor data quality

    Document Center

    DataWorks:Monitor data quality

    Last Updated:Jun 23, 2026

    This topic describes how to use Data Quality to monitor the quality of table data.

    Prerequisites

    Before you begin, complete the tasks described in Synchronize data and Process data.

    • Used Data Integration to synchronize basic user information from the ApsaraDB RDS for MySQL table ods_user_info_d to the MaxCompute table ods_user_info_d.

    • Used Data Integration to synchronize website access logs from the user_log.txt file in OSS to the MaxCompute table ods_raw_log_d.

    • Used Data Studio to transform the collected data into basic user persona data.

    Background

    Data Quality is a one-stop platform that provides quality checks, notifications, and management services for various disparate data sources. Data Quality monitors data in datasets and currently supports MaxCompute tables. When offline MaxCompute data changes, Data Quality checks the data and blocks the production pipeline to prevent the spread of dirty data. Data Quality also provides management of historical check results, which lets you analyze and classify data quality.

    In this tutorial, you use the Data Quality feature of DataWorks to promptly detect changes in the source data and identify dirty data generated during the extract, transform, and load (ETL) process for the user persona analysis case. Based on the user persona analysis and processing flow, the quality monitoring rules are summarized as follows:

    Table name

    Requirement details

    ods_raw_log_d

    Monitor the daily synchronized raw log data to ensure the table is not empty. This prevents invalid processing and mitigates issues promptly.

    ods_user_info_d

    Check user information data that is synchronized daily. Ensure the row count is not zero (strong rule) and the business primary key is unique (weak rule). This prevents invalid processing and stops data issues early.

    dwd_log_info_di

    Not monitored separately.

    dws_user_info_all_di

    Not monitored separately.

    ads_user_info_1d

    Configure a rule to monitor the daily row count of your user data table. This lets you track changes in daily unique visitors (UVs) and promptly understand your application's status.

    Go to the rule configuration page

    1. Log on to the DataWorks console. In the target region, click Data Governance > Data Quality in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Quality.

    2. Go to the rule configuration page for the table.

      In the Data Quality left navigation bar, click Configure Rules > Configure by Table, and locate the target table based on the following parameters.

      • Data source: MaxCompute

      • Database: your current production project (workshop2024_01).

      • Table: In this tutorial, you need to configure data quality monitoring for the ods_raw_log_d, ods_user_info_d, and ads_user_info_1d tables.

    3. In the search results, find the target table and click Rule Management in the Actions column to go to the quality details page for the table. Refer to the following sections for specific configurations.

    Configure data quality monitoring rules

    Configure monitoring rules for ods_raw_log_d

    The ods_raw_log_d table receives website access records synchronized from OSS. Based on the business attributes of this table, you configure a row count greater than 0 check and trigger the data quality rule check by associating a monitor.

    1. Configure the monitor

    The monitor checks whether the data in the specified data range (partition) of the target table meets expectations.

    In this step, you need to set Data Range to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether the data meets expectations.

    This means that every time the scheduled task of the ods_raw_log_d table runs, the monitor is triggered and uses the associated quality rules to check the data in the corresponding data range to determine whether it complies with the data quality rules.

    Perform the following steps:

    1. On the quality details page of the table, click the Monitor tab, and then click Create Monitor.

    2. Configure the monitor settings.

      Key parameters:

      Parameter

      Example

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by production scheduling. Select the ods_raw_log_d node created in Synchronize data.

      Monitoring Rule

      Not configured for now. This will be configured in a later section.

      Note

      For more information about configuring data quality monitoring, see Configure monitoring rules.

    This tutorial monitors whether the table data generated by each daily scheduled task meets expectations. Because the table generates data for the previous business date each day, the result is as expected when the scheduling time is today and the calculated result is for yesterday.

    2. Configure monitoring rules

    The ods_raw_log_d table receives website access records synchronized from OSS and is the source table for the user persona analysis scenario. To prevent invalid processing and quality issues in downstream tasks, you configure a strong rule that checks whether the row count of this table is greater than 0, which determines whether the synchronization task has written data to the partition of this table.

    When the monitor detects that the row count of the corresponding partition in the ods_raw_log_d table is 0, an alert is triggered, the ods_raw_log_d node fails, and the downstream tasks of the ods_raw_log_d node are blocked.

    Perform the following steps:

    1. On the Rule Management tab, select the monitor that you created under Monitor Perspective (in this example, raw_log_number_of_table_rows_not_0), and then click the Create Rule button on the right side to go to the Create Rule page.

    2. In System Template, find the The number of rows is greater than 0. rule and click Use. Then set Degree of importance to Strong rules.

      Note

      In this tutorial, the rule is defined as strong, which means that when the monitor detects that the row count of the ods_raw_log_d table is 0, an alert is triggered and downstream tasks are blocked.

    3. Click Determine.

      Note

      For more information about configuration items, see Configure monitoring rules.

    3. Test run the monitor

    A test run verifies whether the check rules configured in the monitor are reasonable. To ensure that the quality rules are correctly configured and meet expectations, you can test run the monitor after the rules are created.

    1. On the Rule Management tab, select the monitor that you created under Monitor Perspective (in this example, raw_log_number_of_table_rows_not_0), and then click Test Run from image on the right side to open the Test Run dialog.

    2. In the Test Run dialog, select a Data Timestamp and click Test Run.

    3. After the test run is complete, follow the on-screen instructions and click View Details to check whether the test result is a pass.

    4. Subscribe to the monitor

    Data Quality provides alert capabilities. By subscribing to a monitor, you can receive timely notifications about quality check exceptions and take action to ensure that data is produced securely, stably, and on time.

    1. On the Rule Management tab, select the monitor that you created under Monitor Perspective (in this example, raw_log_number_of_table_rows_not_0), and then click Subscribe to Alerts from image on the right side.

    2. Follow the on-screen instructions to add a Notification Method and a Recipient, and then click Save in the Actions column.

    3. After the subscription is configured, in the left navigation bar, click Quality O&M > Monitor, and select My Subscriptions to view and modify the tasks to which you have subscribed.

    Configure monitoring rules for ods_user_info_d

    The ods_user_info_d table receives basic user information synchronized from RDS MySQL. Based on the business attributes of this table, you configure a row count greater than 0 check and a business primary key uniqueness check, and trigger the data quality rule check by associating a monitor.

    1. Configure the monitor

    The monitor checks whether the data in the specified data range (partition) of the target table meets expectations.

    In this step, you need to set Data Range to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether the data meets expectations.

    This means that every time the scheduled task of the ods_user_info_d table runs, the monitor is triggered and uses the associated quality rules to check the data in the corresponding data range to determine whether it complies with the data quality rules.

    Perform the following steps:

    1. On the Monitor tab, click Create Monitor.

    2. Configure the monitor settings.

      Key parameters:

      Parameter

      Example

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by production scheduling. Select the ods_user_info_d node created in Synchronize data.

      Monitoring Rule

      Not configured for now. This will be configured in a later section.

      Note

      For more information about configuring data quality monitoring, see Configure monitoring rules.

    2. Configure monitoring rules

    The ods_user_info_d table receives basic user information synchronized from RDS MySQL and is the source table for the user persona analysis scenario. To prevent invalid processing and quality issues in downstream tasks, you configure a strong rule that checks whether the row count of this table is greater than 0, which determines whether the synchronization task has written data to the partition of this table.

    When the rule takes effect and the monitor detects that the row count of the corresponding partition in the ods_user_info_d table is 0, an alert is triggered, the ods_user_info_d node fails, and the downstream tasks of the ods_user_info_d node are blocked.

    Perform the following steps:

    1. On the Rule Management tab, select the monitor that you created under Monitor Perspective (in this example, user_info_quality_control), and then click Create Rule to go to the Create Rule page.

    2. In System Template, find the The number of rows is greater than 0. rule and click Use. Then set Degree of importance to Strong rules.

      Note

      In this tutorial, the rule is defined as strong, which means that when the monitor detects that the row count of the ods_user_info_d table is 0, an alert is triggered and downstream tasks are blocked.

    3. In System Template, find the Unique Value Count, Fixed Value rule and click Use. Then modify Rule Scope, Monitoring Threshold, and Degree of importance as follows.

      • Rule Scope: uid(STRING)

      • Monitoring Threshold: Normal threshold = 0

      • Degree of importance: Weak rule

    4. Click Determine.

      Note

      For more information about configuration items, see Configure monitoring rules.

    3. Others

    The test run and subscription procedures are the same as those in Configure monitoring rules for ods_raw_log_d.

    Configure monitoring rules for ads_user_info_1d

    The ads_user_info_1d table is the final result table. Based on the business attributes of this table, you monitor the row count fluctuation and check the primary key uniqueness to track daily UV fluctuations and promptly understand online traffic changes. The data quality rule check is triggered by associating a monitor.

    1. Configure the partition expression

    The monitor checks whether the data in the specified data range (partition) of the target table meets expectations.

    In this step, you need to set Data Range to dt=$[yyyymmdd-1]. When the monitor runs, it matches the data in this partition to determine whether the data meets expectations.

    This means that every time the scheduled task of the ads_user_info_1d table runs, the monitor is triggered and uses the associated quality rules to check the data in the corresponding data range to determine whether it complies with the data quality rules.

    Perform the following steps:

    1. On the Monitor tab, click Create Monitor.

    2. Configure the monitor settings.

      Key parameters:

      Parameter

      Example

      Data Range

      dt=$[yyyymmdd-1]

      Trigger Method

      Triggered by production scheduling. Select the ads_user_info_1d node created in Process data.

      Monitoring Rule

      Not configured for now. This will be configured in a later section.

      Note

      For more information about configuring data quality monitoring, see Configure monitoring rules.

    2. Create monitoring rules

    The ads_user_info_1d table is used for user persona analysis. To detect daily UV fluctuations, you configure row count fluctuation monitoring and primary key uniqueness checks on the aggregated data to track daily UV changes and promptly understand online traffic fluctuations.

    When the rule takes effect and the monitor detects that the primary key of the ads_user_info_1d table is not unique, an alert is triggered. When the 7-day row count fluctuation rate is greater than 10% and less than 50%, an orange alert is triggered. When the 7-day row count fluctuation rate is greater than 50%, a red alert is triggered.

    Note

    The monitor is configured with the following Disposal Policy:

    • Strong rule · Red alert: The disposal policy is Blocks. When a data quality issue is detected, the system identifies the production scheduling node that triggered the quality check for this table and sets the node to failed. Downstream nodes will not be executed, which blocks the production pipeline to prevent the spread of dirty data.

    • For other alerts, the disposal policy is Alert. When a data quality issue is detected, an alert notification is sent through the alert subscription channel of the monitor.

    Therefore, when you configure quality rules:

    • If you set a strong rule, a red alert blocks downstream task nodes, and other alerts do not block downstream task nodes.

    • If you set a weak rule, a red alert does not block downstream task nodes, and other alerts do not block downstream task nodes.

    Perform the following steps:

    1. On the Rule Management tab, select the monitor that you created under Monitor Perspective (in this example, ads_user_info_quality_control), and then click Create Rule to go to the Create Rule page.

    2. In System Template, find the Table Row Count, 7-Day Fluctuation Rate rule and click Use. Then modify Monitoring Range and Degree of importance as follows.

      • Monitoring Threshold:

        • Red threshold > 50%

        • Orange threshold > 10%

        • Normal threshold <= 10%

      • Degree of importance: Weak rule

    3. In System Template, find the The number of rows is greater than 0. rule and click Use. Then set Degree of importance to Strong rules.

    4. Click Determine.

      Note

      For more information about configuration items, see Configure monitoring rules.

    3. Others

    The test run and subscription procedures are the same as those in Configure monitoring rules for ods_raw_log_d.

    Next step

    After data processing is complete, you can use the Data Analysis module to visualize the data. For more information, see Visualize data.