You can add monitored objects and perform operations such as viewing, editing, and deleting monitored objects on the rule list page. This topic describes how to add and manage data table quality rules.
Limits
When tables with the same name but different letter cases exist in the same project or database, only one of them can be added as a monitored object.
You can select a maximum of 200 objects when adding monitored objects in batch.
Only Dataphin data tables and global data tables support quality score weight configuration.
The quality score weight of a monitored object is used to calculate the quality score from perspectives such as project, data source, and individual.
Permission description
Super administrators and quality administrators can add monitored objects.
You can only export quality rules for monitored objects that you have permission to view.
The operation permissions for different objects vary. For more information, see Quality rule operation permissions.
View quality rules from different perspectives
Dataphin supports viewing data table quality rules from three perspectives: Global, Project, and Individual.
Global: Super administrators, quality administrators, and custom global roles with Quality Rule-Management permission can view all data table quality rules from the global perspective.
Project: Super administrators and quality administrators can switch between all projects to view data table quality rules. Project members can only select projects they have joined and view quality rules for data tables in those projects.
Individual: Super administrators and quality administrators can switch to any member to view data table quality rules. Regular members can only view quality rules for Dataphin data tables for which they are the quality owner and cannot switch to other members.
If permission restrictions are enabled in quality rules, the permission restrictions of the quality rules take precedence.
When viewing quality rules from the project perspective, only monitored objects and quality rules in the specified project are displayed.
When viewing quality rules from the individual perspective, only monitored objects and quality rules for which you are the quality owner are displayed.
Add a monitored object
On the Dataphin homepage, choose Administration > Data Quality from the top navigation bar.
In the left navigation bar, choose Quality Monitoring > Quality Rule. On the Quality Rule page, click Add Monitored Object in the upper-right corner.
In the Add Monitored Object dialog box, filter monitored objects based on different conditions.
Dataphin Table: Filter data tables by project (for physical tables) or business unit (for logical tables).
Global Data Table: Filter data tables by data source type, data source, and DB/Schema. For supported data sources, see Data sources supported by Dataphin. If a data source cannot connect to the Dataphin cluster, you need to perform metadata acquisition before configuring quality monitoring rules. For more information, see Create and manage metadata acquisition tasks.
Metric: Filter metrics by business unit and logical aggregate table.
Data Source: Filter data sources by data source type. You can select all data sources in Dataphin to create quality monitoring rules. All supported data sources can be tested for connectivity, but only some data sources support table structure change monitoring quality rules. For more information, see Data sources supported by Dataphin.
Real-time Dimension Table: Filter data tables by environment and project.
Click Configure Quality Rules to go to the quality rule configuration page.
Manage the monitored object list
On the Quality Rule list page, you can view quality rule information for configured monitored objects, such as table name, project/business unit, number of effective/total quality rules, and perform operations such as viewing, running, managing quality owners, and deleting. The page display is consistent for different monitored objects. The following example uses Dataphin Table.
Operation | Description |
① Individual operations | |
View Quality Rule Details | View the configuration details of data table quality rules. |
Run | Run the data table based on the configured schedule or custom partition. |
View Quality Report | View the quality report for the data table. |
Quality Owner Management | Enter the quality owners to whom you want to transfer ownership. You can select up to 20 owners. |
Edit Quality Score Weight | Modify the quality score weight. Important
|
Delete | Deleting this quality rule object will delete all quality rules under the object. This operation cannot be undone. |
② Batch operations | |
Run | Run data tables in batch based on the configured schedule or custom partition. For non-partitioned tables, you can choose to validate the entire table. Note
|
Batch Manage Quality Owners | Support batch Append or Modify quality owners.
|
Export Rules | Export custom SQL quality rules that you have permission to view for the monitored objects selected on the current page. |
Enable/Disable Quality Rule Validation | You can enable or disable quality rule validation in batch. |
Edit Quality Score Weight | Modify quality score weights in batch. Important
|
Alert Configuration | You can configure alert methods for quality rules under different monitored objects in batch. If an alert with the same monitoring scope has already been configured for the selected monitored object, the latest configuration will overwrite the existing one. If there is no alert configuration with the same monitoring scope, a new alert configuration will be added. When configuring alerts in batch, the monitoring scope can only be: All Rules, All Hard Rules, or All Soft Rules. The configuration details are the same as for individual object alert configuration. For more information, see Alert configuration. Note
|
Permission Management | You can specify members who can view validation record details, quality rule details, and quality reports. In the Quality Rule Permission Management dialog box, select the members who can view details and click OK. |
Delete | You can delete quality rule objects in batch. This will delete all quality rules under the objects. This operation cannot be undone. |
What to do next
After adding a monitored object, you can configure quality rules for monitoring. For more information, see: