Numeric rules

Configuration example

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: "dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'"
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - "avg(size) between 100 and 300"
  - "duplicate_count(product_id) = 0"
  - "duplicate_percent(number_employees) < 5%"
  - "max(size) <= 500"
  - "min(size) >= 50"
  - "row_count > 0"
  - "sum(discount) < 120"
computeResource:
  id: 2001

Problem data retention

For the duplicate_count, duplicate_percent, distinct_count, and distinct_percent metrics, enable problem data retention to collect the rows that fail the check. Set collectFailedRows: true on the rule object instead of using an inline rule string.

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: "dt='$[yyyymmdd]' AND hh='$[hh24-1/24]'"
    dataSource:
      name: odps_first
      envType: Dev
rules:
  - rule: "duplicate_percent(number_employees) < 5%"
    collectFailedRows: true
computeResource:
  id: 2001

Problem data retention is not available for the other six metrics (avg, row_count, sum, min, max, and table_size).

Metrics

Metric	Description	Example
`avg`	Average value of a column	`avg(size) between 100 and 300`
`row_count`	Total number of rows	`row_count > 0`
`sum`	Sum of all values in a column	`sum(discount) < 120`
`min`	Minimum value in a column	`min(size) >= 50`
`max`	Maximum value in a column	`max(size) <= 500`
`distinct_count`	Number of unique values in a column	`distinct_count(product_id) > 0`
`distinct_percent`	Percentage of unique values relative to total rows	`distinct_percent(number_employees) > 0%`
`table_size`	Size of the table's data storage	`table_size > 0`
`duplicate_count`	Number of rows that contain duplicate values	`duplicate_count(product_id) = 0`
`duplicate_percent`	Percentage of rows with duplicate values relative to total rows	`duplicate_percent(number_employees) < 5%`