Dataphin:Batch create quality rules

Prerequisites

You have published data tables or metrics to the production environment. For more information about publishing, see Manage publishing tasks.

Permission description

• Super administrators and quality administrators can batch configure quality rules, create and delete abnormal archive tables, and configure scoring weights.

• Quality owners can configure quality rules, create and delete abnormal archive tables, and configure scoring weights for the monitored objects they are responsible for.

• Quality owners and regular users need read permissions for data tables and data sources. To apply for permissions, see Apply for, renew, and return table permissions and Apply for data source permissions.

• Operation permissions vary for different objects. For more information, see Quality rule operation permissions.

Validation rule description

When a data table participates in quality rule validation, if a weak monitoring rule is triggered, the system sends you an alert message to help you detect and handle exceptions promptly. If a strong monitoring rule is triggered, the system automatically interrupts the task containing the table to prevent dirty data from flowing downstream, and sends you an alert message to help you detect and handle exceptions promptly.

Batch add quality rules

Batch adding quality rules can meet scenarios where different objects need to be configured with the same quality rule, improving rule configuration efficiency. It supports table-level or field-level configuration. The configuration methods for different monitored objects are basically the same except for the object selection method. The following figure shows an example of a Dataphin table.

1. On the Dataphin homepage, click Administration > Data Quality in the top navigation bar.

2. Click Quality Rule in the left-side navigation pane. On the Quality Rules page, click Add Quality Rule in the upper-right corner or click the icon, and then select Add By Monitored Object.

3. On the Add Quality Rule page, configure the parameters.

   1. Basic information configuration

      Parameter	Description
      Rule Name	Customize the name of the quality rule, which cannot exceed 256 characters. After selecting the monitored object, you can adjust the name for each object separately.
      Rule Strength	Supports Weak Rule and Strong Rule. Weak Rule: If you select Weak Rule, when the quality rule validation result is abnormal, an alert is triggered but downstream task nodes are not blocked. Strong Rule: If you select Strong Rule, when the quality rule validation result is abnormal, an alert is triggered. If there are downstream tasks (code check scheduling, task trigger scheduling), they will be blocked to prevent data pollution from spreading. If there are no downstream tasks (such as periodic quality scheduling), only an alert is triggered.
      Description	Customize the description of the quality rule. It cannot exceed 128 characters.
      Configuration Method	Supports template creation and custom SQL. Template Creation: Use common system templates and custom business templates to quickly create quality rules. System Template: Template parameters are configurable, suitable for creating common rules. Custom Template: Template parameters are preset and do not need to be configured, generally used for creating rules with business logic. SQL: You can flexibly customize quality monitoring rules through SQL, suitable for flexible and complex scenarios. Only custom SQL templates support batch configuration of quality rules. Note Data sources and real-time metadata tables do not support configuration methods.
      Rule Template	Different monitored objects support different rule templates. For more information about templates, see Template type description. Dataphin tables and global data tables support rule templates including Completeness, Uniqueness, Timeliness, Validity, Consistency, Stability, and SQL. Completeness: Includes Null Value Validation and Empty String Validation. Uniqueness: Includes Uniqueness Validation, Field Group Count Validation, and Duplicate Value Count Validation. Timeliness: Includes Time Comparison With Expression, Time Interval Comparison, and Time Interval Comparison In Two Tables. Validity: Includes Column Format Validation, Column Length Validation, Column Value Domain Validation, Reference Table Validation, and Standard Reference Table Validation (requires activation of the Data Standard module). Consistency: Includes Columns Value Consistency Validation, Columns Statistical Consistency Validation, Single Field Business Logic Consistency Comparison, Columns In Two Tables Value Consistency Validation, Columns In Two Tables Statistical Consistency Validation, Columns In Two Tables Processing Logic Consistency Validation, and Cross-Source Columns Statistical Consistency Validation. Stability: Includes Table Stability Validation, Table Volatility Validation, Column Stability Validation, and Column Volatility Validation. SQL: Contains information created by custom SQL rule templates. Metrics support rule templates including Uniqueness and Stability. Uniqueness: Includes Field Group Count Validation and Duplicate Value Count Validation. Stability: Includes Column Stability Validation and Column Volatility Validation. Data sources support the rule template Stability. Connectivity Monitoring: Monitors and alerts on connectivity changes when data sources configured in Dataphin cannot be connected due to network changes, username, password, or other reasons, which can cause task errors. Table Structure Change: Monitors and alerts on changes to upstream table structures, such as renaming, deletion, or adding/removing fields, which can cause downstream errors. Only some data sources support configuring table structure change monitoring rules. For more information, see Limits. Real-time metadata tables support rule templates including Consistency and Stability. Consistency: Includes Stream-Batch Comparison and Real-time Link Comparison. Stability: Includes Real-time Statistical Value Detection.
      Rule Type	The rule type is related to the template and is the most basic attribute of the template, which can be used for description and filtering functions.
      Object Filtering	You can filter monitored objects based on different conditions. Dataphin tables: Filter data tables by project (table type is physical table)/business unit (table type is logical table), resource owner, and table type. You can select up to 100 projects/business units. Global data tables: Filter data tables by data source type, data source, and DB/Schema. For supported data sources, see Data sources supported by Dataphin. If the data source cannot connect to the Dataphin cluster, you need to perform metadata acquisition before configuring quality monitoring rules. For supported data sources, see Create and manage metadata acquisition tasks. Metrics: Filter metrics by business unit and logical aggregate table. Data sources: Filter data sources by data source type. You can create quality monitoring rules for all data sources in Dataphin. All supported data sources can undergo connectivity testing, but only some data sources support configuring table structure change monitoring quality rules. For more information, see Data sources supported by Dataphin. Real-time metadata tables: Filter real-time metadata tables by project.
      Object Selection	Field Objects: If you need to configure field-level monitoring rules, you can first select the data tables to be monitored based on table name, table owner, and quality owner, and then select the specific fields to be monitored. Table Objects: When the rule template for data tables is selected as Stability-Table Stability Validation and Stability-Table Volatility Validation, or when the rule template for data sources is selected as Stability-Table Structure Change Monitoring, you can configure table-level monitoring rules. You can select the data tables to be configured based on table name, table owner, and quality owner.

   2. Click Next.

      After clicking Cancel, all quality rules configured in this session will not be added.

   3. Rule configuration (Data sources do not require configuration, you can directly view the next step)

      Parameter	Description
      Reference Table	The data table selected in the object selection. Rule details are configured based on the fields of this table. For example: Table A has two fields, id and name; Table B has two fields, id and age; Table C has two fields, name and age. If Table A is the reference table and id is the validation field, then Table B passes validation, but Table C fails validation. Note When the monitored object is a data table or real-time metadata table and the rule template is selected as complex (i.e., validation requires fields other than the validation field), a reference table needs to be configured. When you need to batch configure comparison fields for different tables with different fields, the reference table can provide quick selection. Reference table usage scenarios: If you have similar or identical requirements, batch configuration is recommended. If the requirements are completely different, using a reference table will definitely result in errors during the third step of validation.
      Template Configuration	When you select a quality rule template, the template's configuration information is displayed. To modify the configuration information, see Quality rule templates.
      Rule Configuration	Rule configuration varies depending on the selected rule template. For data table rule configuration details, see Data table parameter configuration. For metric rule configuration details, see Metric rule configuration. For real-time metadata table rule configuration details, see Offline link comparison parameter configuration and Multi-link comparison parameter configuration. Special configurations are as follows: Validation Table Data Filtering: Disabled by default. When enabled, you can configure filter conditions, partition filtering, or regular data filtering for the validation table. The filter conditions will be directly appended to the validation SQL. If the validation table requires partition filtering, it is recommended to configure the partition filter expression in the scheduling configuration. After configuration, the quality report will be viewed with the validation partition as the minimum granularity. When the rule template is selected as Consistency/Columns In Two Tables Statistical Consistency Validation or Consistency/Cross-Source Columns Statistical Consistency Validation, you can choose whether to enable Comparison Table Data Filtering. When enabled, you can configure filter conditions, partition filtering, or regular data filtering for the comparison table. The filter conditions will be directly appended to the validation SQL.
      Validation Configuration	After data quality rule validation, the results are compared with the abnormal validation configuration. If the conditions are met, the validation result is considered failed, triggering alerts and subsequent processes. The available metrics for abnormal validation are determined by the template and configuration content. Multiple conditions with AND/OR operators are supported, but it is recommended to use fewer than 3 conditions in actual configuration. For more information, see Validation configuration description.
      Archive Configuration	Disabled by Default. When enabled, abnormal data can be archived to files or archive tables. After quality validation, you can download and analyze the archived abnormal data. For more information, see Abnormal archiving. Note Only Dataphin tables and global data tables support configuring abnormal archiving.
      Business Property Configuration	The business property input format depends on the configuration of the quality rule property. For example, if the field value type for the rule owner property is custom input with a property field length of 256, then when creating a quality rule, the property value can be input with up to 256 characters. For more information, see Property information.
      Quality Score Configuration	Quality validation results are evaluated using quality scores to help you understand data quality. For configuration details, see Quality score configuration. Note Only Dataphin tables and global data tables support this configuration.

   4. Click Next.

   5. Object details configuration

      • Field-level details configuration

        You can view the field validation information of the selected data tables, and you can also modify the rule name, quality owner, and quality score weight (only supported for Dataphin tables and global data tables). Additionally, you can edit quality rules, delete validation objects, batch modify quality owners, and batch edit quality score weights for validation objects.

        

        • Modify quality owner: Quickly modify the quality owner of selected monitored objects. You can also Batch Manage Quality Owners, supporting batch Append or Modify.

          Note

          If the current validation object has already been added as a monitored object, after successfully batch adding quality rules, the quality owner configured here will overwrite and update the existing quality owner of that monitored object.

          • Append: If there are already 20 quality owners in the current quality management list, no more can be added.

          • Modify: You can modify all quality owners in the current quality management list to the specified owners in this operation, selecting up to 20 owners.

        • Modify quality score weight: Quickly configure the quality score weight of quality rules, used for calculating the quality score of monitored objects. Supports configuring integers between 1 and 10.

        • Edit: Modify the rule configuration, validation configuration, business property configuration, and quality score configuration information of the rule.

        • Delete: Delete the validation object.

      • Table-level details configuration

        You can view the validation information of the selected data tables, and you can also modify the rule name, quality owner, and quality score weight (only supported for Dataphin tables and global data tables). Additionally, you can edit quality rules, delete validation objects, batch modify quality owners, and batch edit quality score weights for validation objects. Table-level configuration operations are the same as field-level operations. For more information, see Field-level details configuration.

        

   6. Click Add Rule to complete the configuration.

Batch add quality rules list

After batch quality rules are created, you can edit, trial run, configure scheduling, delete, and perform other operations in the rule configuration list.

What to do next

In the quality rule list, after configuring the schedule, click Complete to view it on the Dataphin table rule list page. For more information, see View monitored object list.