If a node triggers a Data Quality check, you can create rules to denoise the detected abnormal data in the current workspace. This way, alerts are not sent for the data and related nodes are not blocked. In other words, related nodes do not fail to be run due to the failed check.

Prerequisites

Monitoring rules are created in Data Quality. For more information, see Configure monitoring rules by table and Configure monitoring rules based on a monitoring rule template.

Limits

  • Only Alibaba Cloud accounts can create and manage noise reduction rules.
  • A noise reduction rule takes effect only on the monitoring rule that you specify in the current workspace.
  1. Go to the Noise Reduction Management page.
    1. Go to the Overview page of Data Quality. For more information, see Go to the Overview page.
    2. In the left-side navigation pane, click Noise Reduction Management.
      Noise Reduction Management page
  2. Create a noise reduction rule.
    1. Click Create Denoise Rule in the upper-right corner.
    2. In the Create Denoise Rule panel, click Add Denoise Rule. The following table describes the required parameters.
      Add Denoise Rule panel
      ParameterDescription
      Business DateThe date that corresponds to the data timestamp of the partition for which the noise reduction rule takes effect. After the noise reduction rule is enabled, the related node is not blocked if the data in this partition fails to meet your expectations.
      Rule TypeThe type of the monitoring rule for which the noise reduction rule takes effect. You can select one or more rule types from the drop-down list.
      Expected ValueYou can use the data in the partition as the sample data or baseline data to denoise data based on the specified types of monitoring rules in the workspace. Valid values:
      • Denoise Sample: The related node is not blocked if the sample data in the specified partition is detected as abnormal based on monitoring rules. In other words, if the data in the specified partition does not meet your expectations, the related node is still run as expected.
      • Denoise Baseline: The related node that is run after the specified date is not blocked if the data in the specified partition is used as the baseline data and does not meet your expectations. In other words, the related node is still run as expected.
      Note
      • Sample

        The sample value for the current day. For example, if you want to check the fluctuation of table rows on an SQL node in a day, the sample is the number of table rows on that day.

      • Baseline
        The comparison value from the previous N days. Examples:
        • If you want to check the fluctuation of table rows on an SQL node in a day, the baseline is the number of table rows on the previous day.
        • If you want to check the average fluctuation of table rows on an SQL node in seven days, the baseline is the average number of table rows in the last seven days.
      EnabledSpecifies whether to enable the noise reduction rule. After this rule is enabled, the related node is not blocked if the data fails the Data Quality check.
    3. Click Save.

Manage noise reduction rules

On the Noise Reduction Management page, you can view the details of the created noise reduction rule and modify or delete the rule. Noise Reduction Management page