Restrictions

DataQualityScan Spec restrictions

Datasets

Only one dataset is allowed.
Only one table is supported. Wildcard characters are not supported.
Use a WHERE clause in the filter field to specify the timestamp range.

Note
Always specify a filter for a MaxCompute partitioned table. Otherwise, the execution fails.

datasets:
  - type: Table
    tables:
      - ods_d_dq_openapi_log  # Maximum of 1. Wildcard characters are not supported.
    dataSource:
      name: odps_first
      envType: Dev
    filter: "dt = '$[yyyymmdd-1]'"  # Only timestamp ranges specified by a WHERE clause are supported.
rules:
  # ...

Rules

The following threshold formats are supported for fluctuation rules:

Scenario	Requirement	Example
Alert on increase only	`fail` threshold must be greater than `warn` threshold. Both must be greater than 0.	`warn: when > 0.1%` / `fail: when > 0.5%`
Alert on decrease only	`fail` threshold must be less than `warn` threshold. Both must be less than 0.	`warn: when < -0.1%` / `fail: when < -0.5%`
Alert on both increase and decrease	Use `not between` only. The upper and lower bounds must be opposite numbers. The absolute value of the `fail` bounds must be greater than the absolute value of the `warn` bounds. Neither bound can be 0.	`warn: when not between -0.1% and 0.1%` / `fail: when not between -0.5% and 0.5%`

DataQualityTemplate Spec restrictions

The assertion parameter in a rule template accepts only fixed combinations. The following combinations are supported:

Rule	Example `assertion` value
Compare with a static field	`assertion: "row_count > 0"`
7-day average fluctuation	`assertion: "change avg last 7 days percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
30-day average fluctuation	`assertion: "change avg last 1 month percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
1-day epoch fluctuation	`assertion: "change 1 day ago percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
7-day epoch fluctuation	`assertion: "change 7 day ago percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
30-day epoch fluctuation	`assertion: "change 1 month ago percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
7-day variance fluctuation	`assertion: "change var last 7 days percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
30-day variance fluctuation	`assertion: "change var last 1 month percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
1, 7, and 30-day fluctuation detection	`assertion: "change 1 day ago and 7 days ago and 1 month ago percent for avg(size)"` `warn: "when > 0.1%"` `fail: "when > 0.5%"`
Intelligent dynamic threshold	`assertion: "anomaly detection for avg(size)"`

DataQualityAlertRule restrictions

A DataQualityAlertRule can monitor only one DataQualityScan.
All alert rules that monitor the same DataQualityScan must use the same Condition expression.
Only one Condition format is supported. The expression must be constructed using the results status property and the severity property of the corresponding rule in results. These two properties must always appear together.

The Condition expression has the following structure:

results.any { r ->
    r.status == 'Fail'  && r.rule.severity == 'Normal' ||
    r.status == 'Error' && r.rule.severity == 'High'   ||
    r.status == 'Warn'  && r.rule.severity == 'High'
}