Dataphin lets you batch import quality rules using an Excel file. This improves the efficiency of configuring quality rules for multiple data tables. This topic describes how to batch import quality rules.
Prerequisites
The data tables must be published to the production environment. For more information, see Manage publishing tasks.
Limits
You can import quality rules for Dataphin tables and global data tables using an Excel file.
You can only batch import custom SQL quality rules.
Permissions
Quality administrators can configure quality rules for all monitored objects.
Quality owners can configure quality rules only for the monitored objects that they own.
Batch import quality rules
On the Dataphin home page, choose Administration > Data Quality from the menu bar.
In the navigation pane on the left, click Quality Rule. On the Quality Rule page, in the upper-right corner, click the
icon next to the Add Quality Rule button and select Batch Import Quality Rules from the drop-down list.In the Batch Import Quality Rules dialog box, you can upload the file and click Start Validation.
Parameter
Description
STEP 1: Download The Upload Template
If you have a template, upload the configuration file and start validation. If you do not have a template, click to download the quality rule template (.xlsx). You can also click Export Quality Rules to modify and re-upload existing rules.
NoteYou must have the permissions to configure quality rules to upload Excel files.
STEP 2: Upload The Excel Record File
After you add the data tables and rules to the template, upload the Excel file. You can upload only one file at a time. The format of the Excel file template is similar to the configuration of a custom SQL system template for a single table. For more information, see Configure quality rules.
In addition to the preceding configuration items, configure the following parameters:
Rule Template Source: Required. An error is reported if you leave this parameter empty.
You can set this parameter to System or Custom. If you select System, the system parses and validates the values in the custom SQL field. If you select Custom, the system does not parse the values in the custom SQL field because they are already configured in the rule template.
Rule Template Type: Optional. Select a template type. The options are custom statistical metric check and custom data detail check.
Rule Configuration: This parameter is required when the rule template source is custom and the rule template contains variables.
Assign values to all variables except t1.
If the monitored table is a Dataphin table, you can enter another Dataphin table as the value of the table variable in the `project.table` format. If the monitored table is a global data table, you can enter another global data table as the value of the table variable in the `schema.table` format.
For a field variable, use the field name directly as its value.
The variable t1 defaults to the current monitored table. You do not need to enter a value for this variable.
NoteA single Excel file cannot contain more than 1,000 rows or exceed 10 MB in size.
Only .xlsx files are supported.
After you upload the Excel file, select a file compatibility policy and view the validation result.
Parameter
Description
Compatibility Policy
This policy handles conflicts between your upload and existing online records. You can select Overwrite or Skip.
Overwrite: If the uploaded rule has the same table, field, and rule name as an existing rule, the existing rule is overwritten.
Skip: If the uploaded rule has the same table, field, and rule name as an existing rule, the duplicate rule is skipped during the upload.
Validation Results
View information about records that passed validation, failed validation, or are duplicates.
Validation Passed: Displays records that passed the upload validation. The information includes the row number in the original file, the monitored object, and the rule name.
Validation Abnormal: Displays records that caused an error during parsing or validation. The information includes the row number in the original file, the monitored object, the rule name, and an error message.
Duplicate Records: Displays records that are duplicates within the upload or that match existing online records. The information includes the row number in the original file, the monitored object, the rule name, and a duplication message.
Download Validation Results: Click the Download Validation Records button to download the validation records as an Excel file.
Click Upload. On the Add Quality Rule page, you can add schedules, change the effective status, and manage owners. For more information, see Batch add quality rules .
What to do next
After you upload the quality rules, you can view and manage the rules on the Quality Rule page. For more information, see View the list of monitored objects.