Data Quality Spec configuration guide

Basic definitions

YAML example

A simple monitoring rule definition:

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - assertion: row_count > 0

Property description

This example defines a monitoring rule with the following properties:

datasets: The monitored object. Includes three properties:
- type: The monitored object type. Only Table is supported.
- dataSource: The data source of the monitored object, identified by name and envType. Call ListDataSources to get available data source names.
  
  Note
  Only some data sources are supported. Supported data source types.
- tables: The table names to monitor.
  
  Note
  For database-level data sources, use schema.table format to monitor tables in non-default schemas.
rules: The rules that data must satisfy. A monitoring configuration can contain multiple rules.

During monitoring, the system scans the data, calculates each rule's metric value, and compares it against the threshold to determine whether the rule passes.

An assertion statement combines a metric type (e.g., row_count), a comparison symbol (e.g., >), and a threshold (e.g., 0). Check results have three states:
- pass: The metric value is within the threshold range.
- fail: The metric value is outside the threshold range.
- error: An exception occurred during the check, such as a syntax error.
Note
- If you specify a warn-level threshold, the check result may also be warn. Add multi-level threshold definitions.
- View supported built-in metric types for the assertion statement.
- When you create a Data Quality monitoring rule, you can also use templates.

Threshold definition methods

Supported threshold methods in assertion:

Fixed threshold

Compare a metric value against a fixed value. A fixed threshold consists of:

a metric
an argument (optional)
a comparison symbol (optional)
a threshold (optional)

Configuration example

rules:
  # The number of data rows must be greater than 0
  - assertion: row_count > 0
  # The maximum value of the size column must be less than or equal to 500
  - assertion: max(size) <= 500

The four parts of a fixed threshold expression:

Expression part	`row_count > 0`	`max(size) <= 500`
a metric	`row_count`	`max`
an argument (optional)	/	`(size)`
a comparison symbol (optional)	`>`	`<=`
a threshold (optional)	`0`	`500`

All available operators are listed in Supported comparison symbols.

Fluctuation threshold

Compare the current metric value with historical values. For example, require that today's user count differs from yesterday's by no more than 100, or that today's revenue stays within 10% of the 7-day average.

Add change for before the metric to define a fluctuation threshold. Parts:

change (keyword)
an aggregate type (optional)
a time window
percent (keyword, optional)
for (keyword)
a metric
an argument (optional)
a comparison symbol (optional)
a threshold (optional)

The combined expression format is: change [aggregate_type] [time_window] [percent] for metric [argument] [comparison_symbol threshold].

Configuration example

Example 1: Compare the difference with the check result of a specified time window (time_window)

Add a time window (time_window) between change ... for to specify which historical values to compare against:

rules:
  # The difference between the current row count and the row count from 7 days ago must be within 10000 rows
  - assertion: change 7 days ago for row_count < 10000

Example 2: Aggregate the check results of a specified time window before comparing the difference

Add an aggregate type between change and the time window (time_window). The system aggregates check records within the time window and uses the result as the reference value:

rules:
  # The difference between the current row count and the average row count of the last 7 days must be within 10000 rows
  - assertion: change average last 7 days for row_count < 10000

The following two aggregate methods are supported:

avg: average.
var: variance.

Without an aggregate method, the system compares the current value with each historical record individually and uses the most severe status as the final result.

Example 3: Compare the fluctuation percentage with historical check results

Add percent between change ... for to compare the fluctuation percentage against the threshold:

rules:
  # The fluctuation percentage of the current row count compared to the row count from 7 days ago must be within 50%
  - assertion: change 7 days ago percent for row_count < 50%

Append % to the threshold for readability.
Assume the current metric value is c and the previous value is cl. The formula is percent = (c-cl) / cl:
- If cl is 0 and c is also 0, the calculated percentage is 0.
- If cl is 0 and c is not 0, the percentage cannot be calculated, and the check result is error.
The result can be negative. Use the between...and... syntax to define a fluctuation percentage range.

Range threshold

You can use between...and... to define a range-type threshold.

Closed range threshold definition

By default, between...and... defines a closed range. The rule below triggers warn when the row count fluctuation exceeds [-1%, 1%] and fail when it exceeds [-5%, 5%]. A value in the [10, 15] range passes.

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - assertion: change 1 day ago percent for row_count
    warn: 
      - when not between -1% and 1%
    fail: 
      - when not between -5% and 5%

Add multi-level threshold definitions

Instead of defining a threshold in the assertion statement, you can omit it from assertion and use warn or fail to define multi-level thresholds:

rules:
  - assertion: duplicate_count(phone)
    warn: when between 1 and 10
    fail: when > 10

This defines two threshold levels, warn and fail:

The check passes when the number of duplicate rows in the phone column is 0.
The check fails when the number of duplicate rows in the phone column is greater than 10.
The check result is warn when the number of duplicate rows in the phone column is between 1 and 10.

Handling strategy when warn and fail overlap

If warn and fail ranges overlap, the system uses the more severe status (fail).

Use not between...and... to define the complement of a range

Add not before between...and... to negate the range:

rules:
  - assertion: duplicate_count(phone)
    warn: 
      - when not between -1% and 1%
    fail:
      - when not between -5% and 5%

The resulting range:

Set a rule identity

Specify an identity as a rule's unique identifier within a monitoring configuration.

If no identity is specified, the system auto-assigns an id.
Ensure identity is unique within a monitoring configuration. Duplicate identities may cause update failures or unintended overwrites. Use readable strings for easier management.
For fluctuation threshold (change...for...) and anomaly detection (anomaly detection for ...) rules, the system queries check history by identity.
When updating a monitoring configuration, the system:
- Matches each rule's id against existing rules and updates matches.
- Deletes existing rules not matched by any id in the request.
- Creates unmatched rules from the request as new rules.

identity example:

rules:
  - assertion: row_count > 0
    name: Row count is greater than 0
    # Specify a unique identifier within the Data Quality monitoring configuration
    identity: table-not-empty

Define the rule business severity (severity)

Set severity to indicate a rule's business impact:

rules:
  - assertion: row_count > 0
    severity: High

severity levels:

High
Normal (default)

Set the rule enabled status (enabled)

Set enabled to false to temporarily disable a rule without deleting it. Disabled rules (enabled: false) are skipped during monitoring.

rules:
  - assertion: row_count > 0
    # The enabled status of the rule. Default value: true.
    enabled: false

Set pre-execution statements (settingConfig)

Add settingConfig to execute SET statements before the metric calculation SQL runs:

rules:
  - assertion: row_count > 0
    # Set pre-execution SET statements
    settingConfig: SET odps.sql.udf.timeout=600s; SET odps.sql.python.version=cp27;

Set the failed row collection switch (collectFailedRows)

Enable failed row collection at the rule level. When a check returns warn or fail, the system saves the failing rows to another table in the same database:

rules:
  - assertion: duplicate_count(phone) = 0
    collectFailedRows: true

Set collectFailedRows to true to enable collection. For custom SQL rules, also specify failedRowsQuery with the filter statement:

rules:
  - assertion: id_null_count = 0
    id_null_count:
      expression: id IS NULL
    collectFailedRows: true
    failedRowsQuery: SELECT * FROM tb_d_spec_demo WHERE dt = '$[yyyymmdd-1]' AND id IS NULL

Besides custom SQL rules, only these metrics support collectFailedRows:

missing_count
missing_percent
duplicate_count
duplicate_percent
distinct_count
distinct_percent

Set rule name and description

Set a name and description for each rule and the overall monitoring configuration.

Names appear in check results and management pages.

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - assertion: row_count > 0
    # Rule name and description
    name: Row count is greater than 0
    description: The output data must not be empty

Set data filters (filter)

Set data filters for rules

Add a filter to check only a subset of data, such as rows where id is not NULL:

rules:
  - assertion: row_count > 0
    filter: id IS NOT NULL

Set data filters for monitoring configurations

Add a filter to Scan.Dataset to apply the filter to all rules in the monitoring configuration.

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'
    dataSource:
      name: odps_first
rules:
  - assertion: row_count > 0

With a filter on the dataset, the system applies the filter before each rule check.

Note

The filter uses a time expression $[yyyymmdd-1]. The system substitutes triggerTime with the applied offset into the filter. Parameter reference methods.

Appendix

Supported comparison symbols (comparison symbol)

>
>=
<
<=
=
!=
between ... and ...

Supported data source types

Supported data source types:

maxcompute
hologres
emr
mysql
analyticdb_for_mysql
analyticdb_for_postgresql
cdh
starrocks

Supported built-in metric types

avg
row_count
sum
min
max
distinct_count
distinct_percent
table_size
missing_count
missing_percent
duplicate_percent
duplicate_count
group_by
invalid_count
invalid_distinct_count

Time window (time_window) definition methods

Supported time window formats:

n time units ago: n (minute[s]|hour[s]|day[s]|week[s]|month[s]) ago, for example, n months ago, n days ago, or n hours ago.
- 1 day ago: 1 day ago
- 7 days ago: 7 days ago
- 1 month ago: 1 month ago
- 8 hours ago: 8 hours ago
- 15 minutes ago: 15 minutes ago
Last n time units: last n (minute[s]|hour[s]|day[s]|week[s]|month[s]) , for example, last 15 minutes, last 7 days, or last 1 month.
- Last 15 minutes: last 15 minutes
- Last 24 hours: last 24 hours
  
  Note
  Collects a data point at each one-hour interval, starting 24 hours before the current time.
- Last 7 days: last 7 days
- Last 1 month: last 1 month
A specific day of a month or a specific day of a week at the same time: 1/2/3/.../-3/-2/-1 of (current|last|n) (months|weeks) (ago)
- The 1st day of the current month at the same time: 1 of current month
- The last day of the previous month at the same time: -1 of last month
- Tuesday of 3 weeks ago at the same time: 2 of 3 weeks ago