Dataphin:Create a Dataphin Table Quality Rule

Prerequisites

You must add a monitored object before configuring quality rules. For instructions, see Add a Monitored Object.

Permissions

• Super administrators, quality administrators, custom global roles with the Quality Rule Manage permission, and custom project roles with the Project Quality Management Quality Rule Management permission for the project where the table resides can configure scheduling, alerting, exception archiving tables, and scoring weights for quality rules.

• A quality owner can configure scheduling, alerting, exception archiving tables, and scoring weights for quality rules on monitored objects they own.

• Quality owners and regular users require read permissions on the Dataphin table. To request these permissions, see Request, Renew, or Release Table Permissions.

• Supported operations vary by object type. For details, see Quality Rule Operation Permissions.

Quality rules configuration instructions

Quality rules support two configuration methods: Custom Configuration and From Standard (this method requires you to enable the data standard module).

• Custom Configuration uses built-in or custom quality rule templates to quickly create rules and supports custom SQL to meet flexible monitoring requirements.

• From Standard references quality rules already configured in the data standard mapped to the current asset to better enforce standard constraints.

Validation Rule Reference

If a quality rule triggers a soft rule, Dataphin sends an alert so you can detect and fix issues quickly. If it triggers a strong rule, Dataphin stops the task that writes to the table to prevent dirty data from flowing downstream. Dataphin also sends an alert so you can detect and fix issues quickly.

Difference Between Trial Run and Run

Trial run and run differ in execution method and result display. A trial run simulates one-time execution of a quality rule to test its correctness and behavior. Trial run results do not appear in quality reports. A run checks the quality rule at a scheduled time. Run results appear in quality reports so you can view and analyze them.

Configure a Quality Rule

1. In Dataphin, click Governance > Data Quality in the top menu bar.

2. In the navigation pane on the left, click Quality Rule. On the Dataphin Table page, click the name of the target object to open the Quality Rule Details page. Configure the quality rule here.

   • Custom Configuration

     1. Hover over Create Quality Rule on the data table and select Custom Configuration. Or click Create Quality Rule to open the Create Quality Rule dialog box.

     2. In the Create Quality Rule dialog box, configure the parameters.

        Parameter		Description
        Basic Information	Rule Name	The name of the quality rule. Up to 256 characters.
        	Rule Strength	Supports Soft Rules and Strong Rules. Weak Rule: If you choose Weak Rule, Dataphin sends an alert when verification fails. But it does not block downstream task nodes. Strong Rule: If you choose Strong Rule, Dataphin sends an alert when verification fails. It also blocks downstream tasks if they exist (for code check scheduling or task-triggered scheduling). This prevents dirty data from spreading. If no downstream tasks exist (such as periodic quality scheduling), Dataphin only sends an alert.
        	Description	A description of the quality rule. Up to 128 characters.
        	Configuration Method	Template Creation: Use common system templates or custom business templates to create quality rules quickly. System Template: Built-in parameters are configurable. Best for general rule creation. Custom Template: Predefined parameters require no configuration. Best for rules with business logic. SQL: Use SQL to define flexible quality monitoring rules. Best for complex scenarios.
        	Rule Template	Select a rule template from the drop-down list: Completeness, Uniqueness, Timeliness, Validity, Consistency, Stability, or SQL. Completeness: Includes Null Value Validation and Empty String Validation. Uniqueness: Includes Uniqueness Validation, Field Group Count Validation, and Duplicate Value Count Validation. Timeliness: Includes Time Comparison With Expression, Time Interval Comparison, and Time Interval Comparison In Two Tables. Validity: Includes Column Format Validation, Column Length Validation, Column Value Domain Validation, Reference Table Validation, and Standard Reference Table Validation (requires Data Standard module). Consistency: Includes Columns Value Consistency Validation, Columns Statistical Consistency Validation, Single-Field Business Logic Consistency Check, Columns In Two Tables Value Consistency Validation, Columns In Two Tables Statistical Consistency Validation, Columns In Two Tables Processing Logic Consistency Validation, and Cross-Source Columns Statistical Consistency Validation. Stability: Includes Table Stability Validation, Table Volatility Validation, Column Stability Validation, and Column Volatility Validation. SQL: Includes Custom Statistic Validation and Custom Detail Value Validation. For details, see Template Types.
        	Rule Type	The rule type depends on the template. It is a basic property of the template and can be used for description and filtering.
        	Monitoring Granularity	This option appears only when you select Custom SQL. Choose Entire Table or specify specific fields to monitor.
        Template Configuration	Template Information	This section shows the configuration information for the selected template. To change this information, go to Quality Rule Templates.
        Rule Configuration	Rule Configuration	Rule configuration changes based on the selected template. For details, see Table Parameter Configuration. Special configurations include the following: Verify Table Data Filtering: Off by default. When enabled, you can set filter conditions, partition filters, or general data filters for the verification table. These filters append directly to the verification SQL. If your verification table requires partition filtering, we recommend setting the partition filter expression in the scheduling configuration. After you set it, quality reports show data at the partition level. When you select the Consistency/Two-Table Field Statistical Value Consistency Check or Consistency/Cross-Source Two-Table Field Statistical Value Consistency Check template, you can enable Comparison Table Data Filtering. When enabled, you can set filter conditions, partition filters, or general data filters for the comparison table. These filters append directly to the verification SQL.
        Verification Configuration	Rule Verification	After quality rule verification, Dataphin compares the result with the exception verification settings. If the result meets the criteria, the verification fails. Dataphin then triggers alerts and other follow-up actions. The available metrics for exception verification depend on the template and configuration. You can use AND/OR conditions. We recommend using fewer than three conditions. For details, see Verification Configuration.
        Archive Configuration	Abnormal Archive	The default setting is Off. When you enable it, you can archive abnormal data to files or tables. After quality verification, you can download and analyze the archived abnormal data. Archive Mode supports Archive Exception Fields Only and Archive Full Records. Archive Exception Fields Only: Archives only the monitored fields. Use this when a single field fully identifies exception data. Archive Full Records: Archives full records containing exception data. Use this when you need the full record to locate exception data. Note Archiving full records greatly increases data volume. We recommend using Archive Exception Fields Only in most cases. Archive Location supports Default File Server and Archive Table For Exception Data. If you have not created an exception archive table, click Manage Exception Archive Tables to create one. For details, see Add an Exception Archive Table. Default File Server: The system file server configured when Dataphin was deployed. You can download exception data directly from the Verification Records-Verification Details page. Or access the default file server directly. When using the default file server, Dataphin archives up to 100 exception records per verification. This works best for small-data verification. Archive Table For Exception Data: If you want to store more exception data or group exception data from different verifications for analysis, create a custom archive table. Each quality rule can store up to 10,000 exception records per run. You can download exception data quickly from the verification records page. Or access the archive table directly and manage its lifecycle. This gives you more flexibility. Note Dataphin downloads all exception data from all rules in this run. Download size is capped at 10,000 records. To view more data, archive to a custom exception archive table and access it directly. Your exception archive table must meet specific format requirements. Otherwise, write errors may occur. For details, see Add an Exception Archive Table.
        Business Attribute Configuration	Attribute Information	Business attribute input rules depend on the quality rule attribute configuration. For example, the field for the responsible department uses an enumeration (multi-select) type. Valid values include Big Data Department, Business Department, and Technology Department. So when you create a quality rule, this field appears as a multi-select drop-down. Valid values are Big Data Department, Business Department, and Technology Department. The field for the rule owner uses custom input. Its maximum length is 256 characters. So when you create a quality rule, you can enter up to 256 characters. If the attribute field uses a range input method, configure it as follows: Range: Used when valid values are continuous numbers or dates. Select from >, >=, <, or <=. For more attribute configuration options, see Create and Manage Quality Rule Attributes.
        Scheduling Attribute Configuration	Scheduling Method	Select an existing schedule. If you have not decided on a scheduling method yet, create the quality rule first and configure scheduling later. To create a new schedule, see Create a Schedule.
        Quality Score Configuration	Scoring Method	Choose between quality verification status and data compliance ratio. Quality Verification Status: Scores based on the verification status of the most recent successful verification. Pass = 100 points. Fail = 0 points. Data Compliance Ratio: Uses the ratio of normal data in the most recent successful verification as the score. For example, if data format validity is 80%, the quality score is 80. Not all rule templates support both scoring methods. Templates that support only Quality Verification Status include the following: Field group count validation and field duplicate value count validation under Uniqueness rules. Single-table field statistical value consistency check and cross-source two-table field statistical value consistency check under Consistency rules. All Stability rules. Custom statistic metric validation under Custom SQL rules.
        	Quality Score Weight	The weight of the quality rule in calculating the monitored object's quality score. Choose an integer from 1 to 10.

     3. Click OK to finish custom rule configuration.

        Click Preview SQL to compare the current configuration with the last saved one. This helps you review SQL changes.

        Note

        • Preview SQL is unavailable if required fields are incomplete.

        • The left side shows the SQL preview of the last saved configuration. If none exists, it is blank. The right side shows the SQL preview of the current configuration.

• Reference Data Standard Monitoring

  1. Hover over Create Quality Rule on the data table and select From Standard.

  2. In the From Standard dialog box, select the data standard rule to reference. Filter by Validity, Uniqueness, Completeness, or Stability. Or search by object name.

     In the reference data standard rule dialog box, you can rename the rule and toggle its active state. Click the icon under the linked standard to view standard details. Or click the icon in the Actions column to view the quality rule.

     Note

     After referencing, you cannot edit rule details. You can configure scheduling and change rule strength.

  3. Click Add Selected Rules to complete referencing the data standard rule.

Rule Configuration List

After you create a quality rule, you can view, edit, trial run, run, or delete it in the rule configuration list.

Create a Schedule

1. On the Quality Rule Details page, click the Schedule Configuration tab. Then click Create Schedule to open the Create Schedule dialog box.

2. In the Create Schedule dialog box, configure the parameters.

   Parameter	Description
   Schedule Name	A custom schedule name. Up to 64 characters.
   Schedule Type	Choose Recurrency Triggered, Data Update Triggered, or Task Triggered. Recurrency Triggered: Runs quality checks at scheduled times. Best for data with fixed output times. Recurrence: Quality rule runs use compute resources. Avoid concurrent runs of many rules at once. This could affect production tasks. Recurrence types include Day, Week, Month, Hour, and Minute. Rules run on the system time zone (the user center time zone). They do not use the scheduling time zone (set in Management Center > System Settings > Basic Settings) unless the two zones match. Fill in Recommended Time: Click Fill in Recommended Time. Dataphin recommends a time based on the average completion time of the current table's output tasks. Data Update Triggered: Parses whether each code task run updates the verification scope for the current table. Best for tables with irregular modification tasks or critical monitoring needs. Note We recommend selecting the partition updated by the task (non-partitioned tables verify the entire table). Dataphin auto-detects all data changes for verification. This avoids missing any changes. Task Triggered: Runs the quality rule after or before a specified task completes. Supported task types include Engine SQL, offline pipeline, Python, Shell, Virtual, DataX, Spark_jar, Hive_MR, and database SQL nodes. Best for tables with fixed modification tasks. Note Fixed task-triggered scheduling only supports production environment tasks. If you set a strong rule and the scheduling task fails, it may affect production tasks. Proceed with caution based on business needs. Trigger Timing: Choose when to run quality checks. Options include Trigger after all tasks succeed, Trigger after each task succeeds, and Trigger before each task runs. Triggering Task: Project administrators or O&M system roles can select task nodes from production projects. Search by node output name. Or select from recommended or all tasks. Recommended Tasks: Shows lineage tasks where the current table is the output table. Also shows tasks where the node output name equals the module name or project name plus table name. This matches the scope in Asset Catalog - Asset Details - Output Information. All Tasks: Shows all production tasks the current user has O&M permissions for. Note If you choose Trigger after all tasks succeed, select tasks with the same scheduling cycle. Different cycles may delay rule runs and quality report generation. If you choose Trigger before each task runs, recommended tasks show lineage tasks where the current table is an input.
   Schedule Condition	Off by default. When enabled, Dataphin checks if conditions are met before scheduling. Scheduling occurs only if conditions are met. If not, Dataphin skips this schedule. Business Date/Executed On: Available for Recurrency Triggered (not supported for timed scheduling), Data Update Triggered, and Task Triggered. Choose Standard Calendar or Custom Calendar. For custom calendar setup, see Create Public Calendar. If you choose Standard Calendar, options include Month, Week, and Date. See the image below. If you choose Custom Calendar, options include Date Type and Tag. See the image below. Instance Type: Available for Data Update Triggered and Task Triggered. Choose Recurring Instance, Data Backfill Instance, or One-Time Instance. See the image below. Note Configure at least one rule. To add a rule, click + Add Rule. Configure up to 10 schedule conditions. Set relationships between schedule conditions as AND or OR.
   Verification Scope	For timed scheduling or fixed-task-triggered scheduling, you can specify a custom validation scope. For data-update-triggered scheduling, you can specify the partitions updated by the job. Customize the validation scope. Updated Partition: If the verification task updates a partition, Dataphin uses that partition for the task. Note In dynamic partition scenarios, Dataphin may fail to parse partitions. No quality verification occurs. Variability verification rules (e.g., partition size, row count, field statistics) require explicit partitions. They do not support task-updated partition verification scopes. Non-partitioned tables verify the entire table when data updates occur. Custom Verification Scope: Use this for scenarios where Dataphin cannot parse partitions. Specify the verification scope using business date or execution date expressions. Verification Scope Expression: A drop-down with editable text. Enter the scope directly, such as ds='${yyyyMMdd}'. Or select a built-in expression and modify it. For built-in expressions, see Built-in Partition Filter Expressions. Note Use and or or to combine multiple conditions. Example: province="Zhejiang" and ds<=${yyyyMMdd}. If you configure filter conditions in the quality rule, they combine with the verification scope expression using AND. Both conditions apply during verification. Verification scope expressions support full table scans. Note: Full table scans consume large resources. Some systems do not support them. We recommend using partition filter expressions to avoid full table scans. Verification Scope Budget: Defaults to today's business date.

3. Click OK to finish schedule configuration.

Schedule Configuration List

After creating a schedule, you can view, edit, clone, or delete it in the schedule configuration list.

Configure Alerting

Configure different alerting methods for different rules to distinguish alerts. For example, use phone alerts for strong rule exceptions and SMS alerts for soft rule exceptions. If a rule matches multiple alert configurations, set an alert priority policy.

1. On the Quality Rule Details page, click the Alert Configuration tab. Then click Create Alert Configuration to open the Create Alert Configuration dialog box.

2. In the Create Alert Configuration dialog box, configure the parameters.

   Parameter	Description
   Scope	Choose All Rules, All Strong Rules, All Weak Rules, or Custom. Note Per monitored object, you can configure one alert for each of All Rules, All Strong Rules, and All Weak Rules. New rules match alerts based on rule strength. To change an alert, edit the existing configuration. Custom scope supports up to 200 rules from the current monitored object.
   Alert Configuration Name	Names must be unique per monitored object. Up to 256 characters.
   Alert Recipients	Configure alert recipients and alert methods. Choose at least one recipient and one method. Alert Recipients: Choose from custom, on-call schedule, or quality owner. You can configure up to five custom recipients. You can configure up to three on-call schedules. Alert Methods: Choose from phone, email, text message, DingTalk, Lark, WeCom, or custom channel. Manage these methods in Channel Settings.

3. Click OK to finish alert configuration.

Alert Configuration List

After configuring alerts, you can sort, edit, or delete them in the alert configuration list.

Add an Exception Archive Table

An exception archive table stores records of quality rule verification exceptions.

1. On the Quality Rule Details page, click the Archive tab. Then click + Add Exception Archive Table to open the Add Exception Archive Table dialog box.

2. In the Add Exception Archive Table dialog box, configure the parameters.

   Method supports Create Table and Select Existing Table. These methods add special quality verification fields. Exception data does not write to the original data table.

   • Create Table: Customize the table name. It defaults to current_table_name_exception_data and must be in the project or module of the archive table. After creation, Dataphin creates the table in the same database or data source. Use letters, numbers, underscores (_), and periods (.). Up to 128 characters.

     • If the monitored table is a physical table, Dataphin creates the archive table in the same project.

     • If the monitored table is a logical dimension table or logical fact table, Dataphin creates the archive table in the same project by default. You can also specify a project in the same module, such as projectA.table_name.

     • If the monitored table is a logical aggregate table, specify a project in the same module for the archive table name. Otherwise, Dataphin creates it in a project under the monitored table's module.

     • The archive table must include all fields from the quality monitoring table and verification fields. Script format:

       create table current_table_name_exception_data
        (dataphin_quality_tenant_id      varchar(64)   comment 'Tenant ID' , 
         dataphin_quality_rule_id        varchar(64)   comment 'Quality rule ID', 
         dataphin_quality_rule_name      varchar(256)  comment 'Quality rule name', 
         dataphin_quality_column_name    varchar(1024) comment 'Verification field name', 
         dataphin_quality_watch_task_id  varchar(128)  comment 'Monitored object task ID', 
         dataphin_quality_rule_task_id   varchar(64)   comment 'Rule task ID', 
         dataphin_quality_validate_time  varchar(64)   comment 'Quality verification time', 
         dataphin_quality_archive_mode   varchar(32)   comment 'Exception archive mode: ONLY_ERROR_FIELD/FULL_RECORD', 
         dataphin_quality_error_data     string        comment 'Exception data', 
         ljba_id                         bigint        comment  'ljba_primary key', 
         ljb_id                          bigint        comment  'ljb_primary key', 
         col_tinyint                     tinyint       comment 'Field type: TINYINT (lowercase)',
         col_tinyint_02                  tinyint       comment '2',
         col_smallint                    smallint      comment 'Field type: SMALLINT (lowercase)',
         col_smallint_02                 smallint      comment '4',
         col_int                         int           comment 'Field type: INT (lowercase)',
         col_int_02                      int           comment '6',
         col_bigint                      bigint        comment 'Field type: BIGINT (lowercase)',
         col_bigint_02                   bigint        comment '8',
         col_float                       float         comment 'Field type: FLOAT (lowercase)',
         col_float_02                    float         comment '10',
         col_double                      double        comment 'Field type: DOUBLE (lowercase)',
         col_double_02                   double        comment '11',
         col_decimal                     decimal(38,18) comment 'Field type: DECIMAL(38,18) (lowercase)',
         col_decimal_02                  decimal(38,18) comment '12',
         col_varchar                     varchar(500)   comment 'Field type: VARCHAR(500) (lowercase)',
         col_varchar_02                  varchar(500)   comment '13',
         col_char                        char(10)       comment 'Field type: CHAR(10) (lowercase)',
         col_char_02                     char(10)       comment '14',
         col_string                      string         comment 'Field type: STRING (lowercase)',
         col_string_02                   string         comment '15',
         col_date                        date           comment 'Field type: DATE (lowercase)',
         col_date_02                     date           comment '16',
         col_datetime                    datetime       comment 'Field type: DATETIME (lowercase)',
         col_datetime_02                 datetime       comment '17',
         col_timestmap                   timestamp      comment 'Field type: TIMESTAMP (lowercase)',
         col_timestmap_02                timestamp      comment '18',
         col_boolean                     boolean        comment 'Field type: BOOLEAN (lowercase)',
         col_boolean_02                  boolean        comment '19',
         col_binary                      binary         comment 'Field type: BINARY (lowercase)',
         col_binary_02                   binary         comment '20',
         col_array                       array<int>     comment 'Field type: ARRAY<int> (lowercase)',
         col_array_02                    array<string>  comment '21',
         col_map                         map<string,string>  comment 'Field type: MAP<string, string> (lowercase)',
         col_map_02                      map<string,int>     comment '22',
         ds                              string              comment 'Date partition, yyyyMMdd'
        ) 
       partitioned by 
       (dataphin_quality_validate_date string comment 'Verification date (partition field)');

   • Select Existing Table: Choose a table from the same project or data source. The archive table must include all fields from the quality monitoring table and verification fields. Click View Exception Archive Table DDL to see the CREATE TABLE statement. Script format:

     create table current_table_name_exception_data
      (dataphin_quality_tenant_id      varchar(64)   comment 'Tenant ID' , 
       dataphin_quality_rule_id        varchar(64)   comment 'Quality rule ID', 
       dataphin_quality_rule_name      varchar(256)  comment 'Quality rule name', 
       dataphin_quality_column_name    varchar(1024) comment 'Verification field name', 
       dataphin_quality_watch_task_id  varchar(128)  comment 'Monitored object task ID', 
       dataphin_quality_rule_task_id   varchar(64)   comment 'Rule task ID', 
       dataphin_quality_validate_time  varchar(64)   comment 'Quality verification time', 
       dataphin_quality_archive_mode   varchar(32)   comment 'Exception archive mode: ONLY_ERROR_FIELD/FULL_RECORD', 
       dataphin_quality_error_data     string        comment 'Exception data', 
       ljba_id                         bigint        comment  'ljba_primary key', 
       ljb_id                          bigint        comment  'ljb_primary key', 
       col_tinyint                     tinyint       comment 'Field type: TINYINT (lowercase)',
       col_tinyint_02                  tinyint       comment '2',
       col_smallint                    smallint      comment 'Field type: SMALLINT (lowercase)',
       col_smallint_02                 smallint      comment '4',
       col_int                         int           comment 'Field type: INT (lowercase)',
       col_int_02                      int           comment '6',
       col_bigint                      bigint        comment 'Field type: BIGINT (lowercase)',
       col_bigint_02                   bigint        comment '8',
       col_float                       float         comment 'Field type: FLOAT (lowercase)',
       col_float_02                    float         comment '10',
       col_double                      double        comment 'Field type: DOUBLE (lowercase)',
       col_double_02                   double        comment '11',
       col_decimal                     decimal(38,18) comment 'Field type: DECIMAL(38,18) (lowercase)',
       col_decimal_02                  decimal(38,18) comment '12',
       col_varchar                     varchar(500)   comment 'Field type: VARCHAR(500) (lowercase)',
       col_varchar_02                  varchar(500)   comment '13',
       col_char                        char(10)       comment 'Field type: CHAR(10) (lowercase)',
       col_char_02                     char(10)       comment '14',
       col_string                      string         comment 'Field type: STRING (lowercase)',
       col_string_02                   string         comment '15',
       col_date                        date           comment 'Field type: DATE (lowercase)',
       col_date_02                     date           comment '16',
       col_datetime                    datetime       comment 'Field type: DATETIME (lowercase)',
       col_datetime_02                 datetime       comment '17',
       col_timestmap                   timestamp      comment 'Field type: TIMESTAMP (lowercase)',
       col_timestmap_02                timestamp      comment '18',
       col_boolean                     boolean        comment 'Field type: BOOLEAN (lowercase)',
       col_boolean_02                  boolean        comment '19',
       col_binary                      binary         comment 'Field type: BINARY (lowercase)',
       col_binary_02                   binary         comment '20',
       col_array                       array<int>     comment 'Field type: ARRAY<int> (lowercase)',
       col_array_02                    array<string>  comment '21',
       col_map                         map<string,string>  comment 'Field type: MAP<string, string> (lowercase)',
       col_map_02                      map<string,int>     comment '22',
       ds                              string              comment 'Date partition, yyyyMMdd'
      ) 
     partitioned by 
     (dataphin_quality_validate_date string comment 'Verification date (partition field)');

3. Click OK to finish adding the exception archive table.

   Select Automatically Set as Active Archive Table After Creation to make it the default archive table for future quality rule creation.

Viewing the List of Abnormal Archived Tables

After adding, the first table is the active archive table. Click an archive table name to view its schema. You can also set other tables as active or delete them.

• Set as Active Archive Table: If you set a table as active, all quality rules under this monitored object that use a custom exception archive table will archive exception data to this table.

• Delete: Deletes only the reference to the exception archive table. It does not delete the table itself. You can re-add it later if needed.

View Quality Reports

Click Quality Report to view the Rule Verification Overview and Rule Verification Details for the current quality rule.

• Filter verification details quickly by exception result, partition time, or keywords in rule or object names.

• In the verification details list, click the icon in the Actions column to view rule verification details.

• In the verification details list, click the icon in the Actions column to view execution logs.

Configure Quality Rule Permission Management

1. Click Permission Management. Configure View Details to specify who can view verification records, quality rule details, and quality reports.

   View Details: Choose All Members or Members with Project Quality Management Permissions.

2. Click OK to finish permission management configuration.

What to Do Next

After completing the quality rule configuration, view it on the Dataphin table rule list page. For details, see View Monitored Object List.