Validity rules

Use validity rules to check whether data in your tables meets expected formats or value constraints.

How it works

A validity rule applies a metric to a column and compares the result against a threshold. Each rule has two parts:

Metric — what to measure: invalid_count (number of invalid rows) or invalid_percent (percentage of invalid rows)
Valid data definition — what counts as valid, specified in the valid field using format, regex, or values

Rows that do not match the valid definition are counted as invalid. For invalid_percent, the threshold is a percentage value — for example, < 5% means fewer than 5% of all rows are invalid.

Configuration example

The following example defines three validity rules on the tb_d_spec_demo table, one for each definition method:

datasets:
  - type: Table
    tables:
      - tb_d_spec_demo
    filter: dt=$[yyyymmdd]/hh=$[hh24-1/24]
    dataSource:
      name: odps_first
      envType: Dev

rules:
  - invalid_count(email_address) = 0
    valid:
      format: email
  - invalid_percent(last_name) < 5%
    valid:
      regex: "(?:XX)"
  - invalid_count(house_owner_flag) = 0
    valid:
      values:
        - US
        - CN

Define valid data

Use the valid field to specify what counts as valid. Three definition methods are available.

format

Match values against a built-in format. You can specify multiple formats for the same column.

Format	Description	Example value
`email`	Email address	`user@example.com`
`idcard`	ID card number for the Chinese mainland	`110101199001011234`
`mobile`	Mobile phone number for the Chinese mainland	`13812345678`
`phone`	Landline phone number for the Chinese mainland	`010-12345678`
`decimal`	Numeric value	`3.14`
`date_yyyymmdd`	Date in `yyyyMMdd` format	`20240101`
`date_minus_yyyymmdd`	Date in `yyyy-MM-dd` format	`2024-01-01`
`date_and_time_24h`	Date and time in `yyyy-MM-dd hh:mm:ss` format	`2024-01-01 14:30:00`
`date_and_time_milis`	Date and time with milliseconds in `yyyy-MM-dd hh:mm:ss.SSS` format	`2024-01-01 14:30:00.123`
`date_us`	Date in `MM/dd/yy` format	`01/01/24`
`date_cn`	Date in `yyyy年MM月dd日` format	`2024年01月01日`
`date_iso_8601`	ISO 8601 date and time in `yyyy-MM-ddThh:mm:ssTZD` format	`2024-01-01T14:30:00+08:00`
`time_24h`	Time in `hh:mm:ss` format	`14:30:00`
`money_cn`	Chinese yuan amount	`¥1000`
`money_us`	U.S. dollar amount	`$1000`
`money_gb`	British pound amount	`£1000`
`money_eu`	Euro amount	`€1000`
`money_jp`	Japanese yen amount	`¥1000`

regex

Match values against a regular expression.

valid:
  regex: "(?:XX)"

values

Match values against a fixed set of allowed literal values.

valid:
  values:
    - US
    - CN