Dynamic threshold detection removes the need to manually set and maintain fixed thresholds for data quality rules. Instead of defining a static value, the rule learns from its check history and uses a time series forecasting algorithm to predict the acceptable range. If the actual metric value falls within the predicted range, the check passes. If it falls outside, the check fails.
How it works
Each time a rule runs, the result is added to the check history. The algorithm analyzes this history and computes an upper bound and a lower bound — the expected range for the metric. On the next run, the actual metric value is compared against that range.
-
Actual value within the bounds → check status:
pass -
Actual value above the upper bound or below the lower bound → check status:
fail
Thewarnstatus is not supported. Onlypassandfailare available.
Enable dynamic threshold detection
To enable dynamic threshold detection on a rule, add anomaly detection for before the metric type in the assertion field.
datasets:
- type: Table
tables:
- tb_d_spec_demo
filter: dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'
dataSource:
name: odps_first
envType: Dev
rules:
- # Add "anomaly detection for" before the metric type to enable dynamic thresholds.
assertion: anomaly detection for row_countLimitations
Cold-start period
A rule must accumulate at least 21 historical check records before predictions begin. During this period, the check status is always Abnormal. Predictions start automatically once 21 records accumulate.
Prediction window
The algorithm uses a maximum of 60 recent check records. Records beyond the 60 most recent are not used.
Check frequency consistency
Prediction accuracy depends on a consistent check schedule. If records are generated at irregular intervals — for example, some hourly and others weekly — the algorithm cannot detect reliable patterns, and predictions may be inaccurate.