Quality rule template type - Dataphin - Alibaba Cloud Documentation Center

This topic outlines the various template types for data quality rules.

Data table/indicator template type

Template Classification	Template Description
Completeness
Null Value Validation	Validates that a single field is not a null value.
Empty String Validation	Checks if a single field contains an empty string.
Uniqueness
Uniqueness Validation	Ensures the uniqueness of values in a single field.
Field Group Count Validation	Confirms deduplicated values of single field data.
Duplicate Value Count Validation	Identifies duplicate and redundant data in a single field.
Timeliness
Time Comparison With Expression	Compares the timeliness of a single field with the data timestamp.
Time Interval Comparison	Measures the time difference between two columns within the same table.
Time Interval Comparison In Two Tables	Assesses the time difference between two columns across different tables.
Validity
Column Format Validation	Checks the format of a single field using an expression or regular expression.
Column Length Validation	Validates the length of a single field.
Column Value Domain Validation	Ensures the values of a single field fall within a specified range.
Reference Table Validation	Determines if a single field's value exists within the lookup table.
Standard Reference Table Validation	Checks if a single field's value is present in a lookup table, with direct selection from the data standard module.
Consistency
Columns Value Consistency Validation	Compares the initial values of two fields within the same table.
Columns Statistical Consistency Validation	Compares statistical data, such as sum and maximum value, of two fields within the same table.
Single Field Business Logic Consistency Comparison	Validates the correctness of complex business logic involving multiple fields within the same table.
Columns In Two Tables Value Consistency Validation	Compares the initial values of two fields across different tables.
Columns In Two Tables Statistical Consistency Validation	Compares statistical data, such as sum and maximum value, of two fields across different tables.
Columns In Two Tables Processing Logic Consistency Validation	Validates the correctness of complex business logic involving multiple fields across different tables, such as total sales amount = unit price * quantity.
Cross-Source Columns Statistical Consistency Validation	Ensures the correctness of complex business logic for two fields in two tables from different data sources.
Stability
Table Stability Validation	Validates the stability of a table's size and row count by comparing statistical results with static fields.
Table Volatility Validation	Assesses the volatility of a table's size and row count by comparing statistical results with historical data.
Column Stability Validation	Checks the stability of a field's average and maximum values by comparing statistical results with static fields.
Column Volatility Validation	Evaluates the volatility of a field's average and maximum values by comparing statistical results with historical data.
SQL
Custom Statistic Validation	Validates a table's statistical indicators, supporting both static field and volatility comparison methods.
Custom Detail Value Validation	Performs custom validation on detailed table data, enabling the counting of normal and abnormal rows and supporting data archiving for detected anomalies.

Data source template type

Template Classification	Template Description
Stability
Data Source Connectivity Monitoring	Monitors the connectivity of the data source to ensure consistent access.
Table Structure Change Monitoring	Tracks changes in table metadata to promptly detect structural modifications.

Real-time metadata table template type

Template Detail Classification	Description
Consistency
Stream-Batch Comparison	Validates the consistency of real-time and offline data when using the same statistical logic, flagging significant discrepancies for investigation and resolution.
Multiple Stream Links Comparison	Facilitates high availability by building multiple links for swift switching in case of data anomalies. Monitors the progress of data calculations across multiple links to detect issues like data lag and statistical drift, enhancing the quality of real-time data.
Stability
Real-time Statistical Value Detection	Determines the accuracy of indicator values or statistical data in real-time. Allows comparison with static fields or historical data for validation.