Dataphin:Create metric quality rules

Prerequisites

To configure a quality rule, you must first add a monitoring object. Add a monitoring object.

Permissions

• Super administrators, quality administrators, users with a custom global role that grants the Quality Rules-Manage permission, users with a custom project role that grants the Project Quality Management-Quality Rule Management permission for the table's project, and metric business owners can configure scheduling and alerts for quality rules.

• Quality owners and regular users also need read permissions on logical table fields. Apply for, renew, and return table permissions.

• Supported operations vary by object. Operation permissions on quality rules.

Rule checks

A weak rule triggers an alert. A strong rule also blocks the table's populating task to prevent dirty data from flowing downstream. An alert is also sent.

Trial run vs. run

A trial run simulates rule execution to verify correctness. Results are not included in the quality report. A run executes the rule at a specified time and outputs results to the quality report.

Quality rule configuration

1. On the Dataphin homepage, choose Governance > Data Quality in the top navigation bar.

2. In the left-side navigation pane, click Quality Rules. On the Metrics page, click the target object name to open the Quality Rule Details page.

3. On the Quality Rule Details page, click Create Quality Rule.

4. In the Create Quality Rule dialog box, configure the parameters.

   Parameter		Description
   Basic Information	Rule Name	A custom name for the quality rule.
   	Rule Strength	Set the strength to weak rule or strong rule. weak rule: Sends an alert but does not block downstream tasks when the check fails. strong rule: Triggers an alert and blocks downstream tasks (such as code check scheduling or task-triggered scheduling) to prevent dirty data from spreading. If no downstream tasks exist (such as periodic quality scheduling), only an alert is sent.
   	Description	Custom description for the quality rule. Maximum 128 characters.
   	Configuration Method	Create from Template: Create a quality rule from a system template or a custom business template. System Template: For general rules. Built-in parameters are configurable. Custom Template: For rules with business logic. Parameters are pre-configured. Custom SQL: Define a rule with custom SQL for complex scenarios.
   	Rule Template	Select a rule template from the drop-down list. Options include Uniqueness, Stability, and Custom SQL. Uniqueness: Includes Field Group Count Check and Duplicate Field Value Count Check. Stability: Includes Field Stability Check and Field Fluctuation Check. Custom SQL: Includes Custom Statistical Metric Check. For more information, see Quality rule template types.
   	Rule Type	Associated with the template as its basic attribute. Used in descriptions and for filtering.
   Rule Configuration	Rule Configuration	If you set Rule Template to Uniqueness, the following parameters are available. Field Group Count Check/Duplicate Field Value Count Check: Check Table Data Filtering: Disabled by default. When enabled, configure filter conditions (partition or data filters) for the checked table. Dataphin appends this condition to the check query. For partitioned tables, configure a partition expression in the scheduling configuration to view the quality report at partition granularity. Enter the data filtering content. Examples: id = 12 -- For a single table T1.id=12 and T2.name = "John" -- For two tables If you set Rule Template to Stability, the following parameters are available. Field Stability Check/Field Fluctuation Check: Statistical Method: Select a statistical method based on your business scenario. Check Table Data Filtering: Disabled by default. When enabled, configure filter conditions (partition or data filters) for the checked table. Dataphin appends this condition to the check query. For partitioned tables, configure a partition expression in the scheduling configuration to view the quality report at partition granularity. Enter the data filtering content. Examples: id = 12 -- For a single table T1.id=12 and T2.name = "John" -- For two tables If you set Rule Template to Custom SQL, the following parameters are available. Custom Statistical Metric Check: Custom SQL: Supports SELECT statements. The query must include the primary table. Example: select sum(sale) from tableA where ds=${bizdate};
   Check Configuration	Rule Check	The check result is compared against the anomaly check configuration. If conditions are met, the check fails and triggers alerts. Available anomaly check metrics depend on the template and configuration. Combine conditions with AND or OR. Three or fewer conditions recommended. Metric check configuration.
   Business Attribute Configuration	Attribute Information	The quality rule's attribute configuration determines the standards for entering business attributes. Examples: The value type for the field mapped to the responsible department is an enumeration (multi-select). The available values are Big Data Department, Business Department, and Technology Department. When you create a quality rule, this attribute is shown as a multi-select drop-down list with these values. The value type for the field mapped to the rule owner is custom input, and the maximum field length is 256 characters. When you create a quality rule, you can enter up to 256 characters for this attribute. If the input method for an attribute field is Range, configure it as follows: Range: This is typically used for continuous numerical or date values. You can choose from the operators >, >=, <, and <=. For more information about attribute configurations, see Create and manage quality rule attributes.
   Scheduling Attribute Configuration	Scheduling Method	Select an existing schedule, or create the quality rule first and configure scheduling later. Create a schedule.

5. Click Save to complete the rule configuration.

   You can click Preview SQL to compare the current configuration with the last saved version and view SQL changes.

   Note

   • The SQL preview is unavailable because key information is incomplete.

   • The SQL preview on the left shows the last saved configuration. If no configuration was saved, it is empty. The SQL preview on the right shows the current configuration.

   Rule Configuration List

   View configured metric rules and perform operations such as view, edit, trial run, run, and delete.

   

   Area	Description
   ① Filter and Search Area	Allows you to quickly search by object or rule name. Allows you to filter by rule type, rule template, rule strength, trial run status, and effective status. Note If a business attribute of the quality rule is configured to be searchable or filterable and is enabled, you can search or filter by that attribute.
   ② List Area	Displays the object type/name, rule name/ID, trial run status, Effective Status, Rule Type, Rule Template, Rule Strength, Scheduling Type, and related knowledge base document information. Click the icon before the Refresh button to select the columns to display in the rule list. Effective Status: Perform a trial run before enabling a rule. Enable only rules that pass the trial run to prevent incorrect rules from blocking production tasks. After you enable a rule, it runs automatically according to its schedule. After you disable a rule, it no longer runs automatically but can be run manually. Related knowledge base document: Click View Details to see the knowledge base information associated with the rule. This includes the table name, check object, rule, and related knowledge base document. You can also search for, view, edit, or delete the knowledge base document. View the knowledge base.
   ③ Operations Area	Perform operations: view, clone, edit, trial run, run, configure scheduling, associate knowledge base documents, and delete. View: View the details of the rule configuration. Clone: Quickly clone a rule. Edit: Requires a new trial run after editing. Trial run: You can perform a trial run for a rule by using an Existing Schedule or a Custom Check Range. After the trial run, you can click the icon to View trial run logs. Run: You can run a rule by using an Existing Schedule or a Custom Check Range. After the run, you can view the check result in Check Records. Configure Scheduling: In the dialog box, you can filter schedules by scheduling type or search by schedule name. You can also edit the schedule. Associate knowledge base document: After you associate a rule with a knowledge base document, you can view the associated information in Quality Rules and the Governance Workbench. You can select an unassociated knowledge base document. To create one: Create and manage the knowledge base. Delete: Deleting a quality rule object also deletes all of its quality rules. This action cannot be undone. Proceed with caution.
   ④ Batch Operations Area	Batch operations: trial run, run, configure scheduling, enable, disable, modify business attributes, associate knowledge base documents, and delete. Trial run: Perform a batch trial run for rules by using an Existing Schedule or a Custom Check Range. After the trial run, you can click the icon to View trial run logs. Run: Perform a batch run for rules by using an Existing Schedule or a Custom Check Range. After the run, you can view the check result in Check Records. Configure Scheduling: In the dialog box, you can filter schedules by scheduling type or search by schedule name. You can also edit the schedule to configure it for multiple quality rules in a batch. Only the selected rules that are editable on the Quality Rules list page can be modified. Enable: After you enable rules in a batch, they run automatically according to their schedules. Only the selected rules that are editable on the Quality Rules list page can be enabled. Disable: After you disable rules in a batch, they no longer run automatically but can be run manually. Only the selected rules that are editable on the Quality Rules list page can be disabled. Modify Business Attributes: You can modify business attributes in a batch if their corresponding field value type is single-select or multi-select. If the field value type is multi-select, you can append or modify attribute values. If the field value type is single-select, you can directly modify attribute values. Associate knowledge base document: After you associate rules with a knowledge base document, you can view the associated information in Quality Rules and the Governance Workbench. You can associate a knowledge base document with monitoring objects in a batch. To create a knowledge base document: Create and manage the knowledge base. Delete: You can delete quality rule objects in a batch. This action cannot be undone. Proceed with caution. Only the selected rules that are editable on the Quality Rules list page can be deleted.

Create schedule

1. On the Quality Rule Details page, click the Scheduling Configuration tab, and then click Create Schedule to open the Create Schedule dialog box.

2. In the Create Schedule dialog box, configure the parameters.

   Parameter	Description
   Schedule Name	A custom name for the schedule.
   Scheduling Type	You can select time-based scheduling, Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling. Time-based scheduling: Runs periodic quality checks on a set schedule. Best for data generated at fixed intervals. Scheduling Cycle: Running quality rules consumes computing resources. Avoid running multiple rules concurrently to prevent interference with production tasks. The available cycles are Day, Week, Month, Hour, and Minute. If the system time zone (the time zone in the User Center) differs from the scheduling time zone (the time zone configured in Management Center > System Settings > Basic Settings), the rule runs based on the system time zone. Data Update-triggered Scheduling: Checks whether a code task updated the specified check range for the current table. Best for tables with variable modification tasks or critical tables requiring monitoring on every change. Note Set the check range to the partitions updated by the task. For non-partitioned tables, the entire table is checked. The system automatically detects all data changes, ensuring no changes are missed. Fixed Task-triggered Scheduling: Runs the configured quality rule after or before a specified task completes successfully. You can trigger the rule based on tasks from the following node types: SQL, Offline Pipeline, Python, Shell, Virtual, Dlink, Spark_jar, Hive_MR, and Database SQL. This is suitable for tables whose modification tasks are fixed. Note You can select only production environment tasks for fixed task triggers. If a strong rule is configured and the scheduled task check fails, it may affect production tasks. Proceed with caution based on your business requirements. Supported engine types are MaxCompute, E-MapReduce 3.X, E-MapReduce 5.x, CDH 5.X, CDH 6.X, FusionInsight 8.X, AsiaInfo DP 5.3, Cloudera Data Platform 7.x, Lindorm (compute engine), Transwarp TDH 6.x, and Transwarp TDH 9.3.x. Trigger Time: Select when the quality check is triggered. You can choose to Trigger After All Tasks Run Successfully, Trigger After Each Task Runs Successfully, or Trigger Before Each Task Runs. Triggering Task: The following roles can select a task node from a production project to trigger the task. You can also search by the node's output name. Users who are project administrators for Prod/Basic projects, have the Ops system role for a Prod project, have the Developer system role for a Basic project, or have a custom project role with the Project Quality Management-Quality Rule Management permission in a Prod/Basic project can select task nodes from a production project. Users with a custom global role that grants the Quality Rules-Manage permission can select task nodes from all production projects. Note If Trigger Time is Trigger After All Tasks Run Successfully, select tasks with the same scheduling cycle to avoid delayed rule execution and quality check results.
   Scheduling Conditions	Disabled by default. If enabled, the system checks whether scheduling conditions are met before running the rule. The schedule runs only if conditions are met; otherwise, it is skipped. Business Date/Execution Date: If you select time-based scheduling (does not support execution date), Data Update-triggered Scheduling, or Fixed Task-triggered Scheduling, you can configure the date. You can choose Common Calendar or Custom Calendar. To create a custom calendar: Create a public calendar. If you select Common Calendar, the available conditions are Month, Day of Week, and Date. Example: If you select Custom Calendar, the available conditions are Date Type and Tag. Example: Instance Type: If you select Data Update-triggered Scheduling or Fixed Task-triggered Scheduling, you can configure the instance type. Options include Periodic Instance, backfill instance, and Manual Instance. Example: Note You must configure at least one rule. To add a rule, click + Add Rule. Maximum 10 scheduling conditions. The relationship between scheduling conditions can be set to AND or OR.
   Check Range	If the Scheduling Type is time-based scheduling or Fixed Task-triggered Scheduling, you can set the Check Range to Custom Check Range. If the Scheduling Type is Data Update-triggered Scheduling, you can set the Check Range to Task-updated Partition or Custom Check Range. Task-updated Partition: If the triggering task updates a partition, the quality check runs directly on that partition. Note In dynamic partition scenarios, the partition may not be resolved, and no quality check is performed. Fluctuation check rules, such as checking partition size, row count, or field statistics, require a specific partition and do not support the task-updated partition check range. If a non-partitioned table is updated, the entire table is checked. Custom Check Range: For scenarios where the partition cannot be resolved, you can use a custom check range to specify the partition expression based on the business date or execution date. Check range expression: A drop-down list where you can also type. You can directly enter the range to be checked, such as ds='${yyyyMMdd}'. You can also select a built-in partition expression and modify it to quickly configure the setting. Built-in partition expression types. Note If there are multiple conditions, you can connect them with and or or, such as province="Zhejiang" and ds<=${yyyyMMdd}. If a filter condition is configured in the quality rule, the partition expression and the filter condition are combined with an AND operator. The partition expression supports full table scans. Note: Full table scans consume significant resources and some are not supported. Configure a partition expression to avoid full table scans. Check Range Estimate: Defaults to the current business date.

3. Click OK to finish the scheduling configuration.

Scheduling Configuration List

After you create a schedule, you can view, edit, clone, and delete it in the scheduling configuration list.

Alert configuration

Configure different alert methods per rule. For example, use phone calls for strong rule failures and SMS for weak rule failures. If a rule matches multiple alert configurations, set a policy to determine which takes effect.

1. On the Quality Rule Details page, click the Alert Configuration tab, and then click Create Alert Configuration to open the Create Alert Configuration dialog box.

   

2. In the Create Alert Configuration dialog box, configure the parameters.

   Parameter	Description
   Coverage	Select All Rules, All strong rules, All weak rules, or Custom. Note For a single monitoring object, you can create one alert configuration for each of the All Rules, All strong rules, and All weak rules scopes. New rules automatically match the corresponding alert based on their strength. To change one of these alert configurations, you must edit the existing configuration. For a custom scope, you can select up to 200 of the configured rules under the current monitoring object.
   Alert Configuration Name	Must be unique per monitoring object. Maximum 256 characters.
   Alert Recipient	Select at least one alert recipient and one alert method. Alert Recipient: You can select Custom, On-call schedule, or Quality Owner. You can configure up to 5 custom alert recipients and up to 3 on-call schedules. Alert Method: You can select Phone, Email, SMS, DingTalk, Lark, WeCom, or Custom Channel. You can manage these methods by using Configure Channel Settings.

3. Click OK to finish the alert configuration.

Alert Configuration List

After you configure an alert, you can sort, edit, and delete it in the alert configuration list.

View quality report

Click Quality Report to view the Rule Check Overview and Rule Check Details for the current quality rule.

• You can quickly filter the check details by anomaly result, partition time, or by rule or object name keywords.

• In the Operations column of the rule check details list, click the icon to view the rule check details for the quality rule.

• In the Operations column of the rule check details list, click the icon to view the execution log for the quality rule.

Quality rule permissions

1. Click Permission Management and configure Viewable Details to specify which members can view check record details, quality rule details, and the quality report.

   Viewable Details: You can select All Members or Only members with quality management permissions for the current object.

2. Click OK to finish the permission management configuration.

Next steps

After configuration, view your quality rules on the Quality Rules list page. View the monitoring object list.