Triage & Resolve Data Quality Issues in Dataphin - Dataphin - Alibaba Cloud - Dataphin

Instructions for use

The Data Quality module must be enabled.

Permission description

Super administrators and quality administrators can view and manage all problem checklist details. The available operations include initiating remediation, adding items to the whitelist, ignoring abnormal results, and rejecting and reprocessing problems.
Other members can view the problems that they have submitted. They can also perform operations on problems for which they are the owner. The available operations include initiating remediation, adding items to the whitelist, ignoring abnormal results, and rejecting and reprocessing problems.
All users with permission to view objects can add problems.
The supported operation permissions vary based on the object type. For more information, see Problem checklist operation permissions.

View the problem checklist from different perspectives

Dataphin provides three perspectives for viewing problem checklists: Global, Project, and Personal.

Global: Super administrators and quality administrators can view all problem checklists.
Project: Super administrators and quality administrators can switch to any project to view its problem checklist. Other members can only select projects they have joined.

Note
The project perspective does not display global tables, data sources, or tag object types.
Personal: Super administrators and quality administrators can switch to any member's view. Other members can only view checklists for problems they submitted or are responsible for.

Add a problem

On the Dataphin home page, click Administration > Data Quality in the top menu bar.
In the navigation pane on the left, click Problem Checklist. Then, click Add Problem to open the Report Quality Problem dialog box.

In the Report Quality Problem dialog box, configure the parameters.

Parameter	Description
Problem Object	Select the object for which you want to report a problem. Configure the corresponding parameters based on the object type. For data tables and real-time metatables, if no specific field is selected, the problem is recorded at the table granularity. If a specific field is selected, the problem is recorded at the specified field granularity.
Problem Type	Select the type for this problem. If a suitable problem type is not available, contact an administrator with the required permissions to create one. For more information, see Custom problem types.
Problem Description	Provide a brief description of the problem to help the owner understand the background information. The description cannot exceed 1,000 characters.
Attachment	Click the Upload button to select files from your local machine. You can upload a maximum of 5 files for a single problem. Supported file formats include Doc (.doc, .docx, .dot, .dotx, .rtf, .html, .xml, .txt), Excel (.xls, .xlsx, .csv), PDF, and images (.jpg, .png, .jpeg). Each file must be 10 MB or smaller.
Contact Information	If possible, provide your contact information so the problem owner can contact you for more details. Supported contact methods are phone, email, and other. Phone: You can enter numbers, hyphens (-), plus signs (+), half-width parentheses, and spaces. The length cannot exceed 20 characters. Email: Enter an email address. Other: The input cannot exceed 100 characters.

Click OK to add the problem.

View the problem checklist

The problem checklist displays problems from Data Quality verification and problems reported during asset or tag usage. You can initiate remediation, add items to a whitelist, ignore problems, and send notifications.

Parameter	Description
① Legend cards	Displays the Number of pending problems and Total number of problems for each object type, including Dataphin tables, global data tables, metrics, data sources, real-time metatables, and tags. Switch views to see the corresponding problem checklist.
② Search and filter area	Search by the name of the monitored object or quality rule using fuzzy match. For Dataphin tables, global data tables, metrics, data sources, and real-time metatables, you can filter precisely by pending, submitted by me, owned by me, rule severity, problem status, problem type, submission method/submitter, and problem owner. For tags, you can filter precisely by pending, submitted by me, owned by me, problem status, problem type, and problem owner. Custom Issue Type: Click Manage next to the issue type. In the Manage Issue Types dialog box, enter a name for the issue type. The name can be up to 20 characters long. Click OK to create the issue type. System-built-in problem types cannot be edited or deleted. Custom problem types can be edited or deleted. You can create a maximum of 50 problem types.
③ Problem checklist	For system-detected problems from quality rule verification, the list shows the rule name, severity, check scope, and check time. For manually entered problems, it shows the submitter information. Available operations vary by problem status. For more information, see Single operations. For Dataphin tables and global data tables, the list displays the table name, problem object, rule name/ID, problem status, problem type, problem description, remediation flow, rule severity, problem owner, check scope, check time, submission method/submitter, and related knowledge base document information. For metrics, the list displays the metric name, rule name/ID, problem status, problem type, problem description, remediation flow, rule severity, problem owner, check scope, check time, submission method/submitter, and related knowledge base document information. For data sources/real-time metatables, the list displays the data source name/table name, problem object, rule name/ID, problem status, problem type, problem description, remediation flow, rule severity, problem owner, check time, submission method/submitter, and related knowledge base document information. For tags, the list displays the tag name (code), problem status, problem type, problem description, remediation flow, problem owner, and submission method/submitter information. Click the icon before Refresh to select the fields you want to display in the problem checklist.
④ Batch operations area	For system-detected problems, batch operations include ignoring the current abnormal result, canceling ignore, adding to the whitelist, initiating remediation, modifying the problem type, and associating a knowledge base document. For manually entered problems, batch operations include ignoring the current abnormal result, canceling ignore, initiating remediation, marking as resolved, rejecting and reprocessing, modifying the problem type, and changing the problem owner (for tag object types only). Available batch operations vary by problem status. For more information, see Batch Operations.

Operation description

Problem status

Problem Status	Actions
Pending	System-detected problems support View Details, Initiate Remediation, Ignore This Abnormal Result, Notify Problem Owner, Add to Whitelist, Re-verify, Modify Problem Type, View Operation Log, and Associate Knowledge Base Document. Manually entered problems support View Details, Initiate Remediation, Ignore This Abnormal Result, Notify Problem Owner, Change Problem Owner, Mark as Resolved, Reject and Reprocess, Modify Problem Type, Delete Problem, and View Operation Log.
Validating	System-detected problems support View Details, Initiate Remediation, Notify Problem Owner, Add to Whitelist, Modify Problem Type, View Operation Log, and Associate Knowledge Base Document.
Completed	System-detected problems support View Details, Initiate Remediation, Modify Problem Type, View Operation Log, and Associate Knowledge Base Document. Manually entered problems support View Details, Initiate Remediation, Change Problem Owner, Reject and Reprocess, Modify Problem Type, Delete Problem, and View Operation Log.
Ignored	System-detected problems support View Details, Cancel Ignore, Re-verify, Modify Problem Type, View Operation Log, and Associate Knowledge Base Document. Manually entered problems support View Details, Cancel Ignore, Change Problem Owner, Mark as Resolved, Reject and Reprocess, Modify Problem Type, Delete Problem, and View Operation Log.

Single Operation

Action	Description
View Details	For system-detected problems, you can view the rule verification details, operation log, and related knowledge base information for the problem object. Verification Details: Click the icon in the Actions column. In the View Details dialog box, click the Verification Details tab to view the details of the most recent verification for the current problem, including basic information, verification configuration, and verification results. Operation Log: Click the icon in the Actions column. In the View Details dialog box, click the Operation Log tab to view the historical operation records for the current problem. You can also filter operation records by check scope. Related Knowledge Base: Click the icon in the Actions column. In the View Details dialog box, click the Related Knowledge Base tab to view the knowledge base information associated with the rule. You can also search, view, edit, or delete knowledge base documents. For more information, see View a knowledge base. For manually entered problems, you can view the problem details and operation log. Problem Details: Click the icon in the Actions column. In the View Details dialog box, click the Problem Details tab to view the name of the reported problem object, problem type, problem description, uploaded attachments, contact information, submitter, and owner. Operation Log: Click the icon in the Actions column. In the View Details dialog box, click the Operation Log tab to view the operation records for the current problem, including the problem object name, operation type, problem status, operation description, operator, and operation time.
Initiate Remediation	After initiating remediation, you can track the entire remediation flow for the problem. For information about configuring the remediation flow, see Initiate Remediation. For information about how to manage the remediation flow, see View and manage remediation flows. Note Only super administrators, quality administrators, problem owners, and custom global roles with administration workbench flow management permissions can initiate a flow.
Ignore This Abnormal Result	Changes the status of a pending problem to ignored. We recommend ignoring unimportant problems. Example scenarios: a data source connection was temporarily interrupted but is now stable, or an abnormal result occurred in a historical task no longer used in current data processing.
Cancel Ignore	After you cancel the ignore status for a system-detected problem, the problem's status is determined by the latest rule verification result. If the verification passes, the problem status changes to Completed. If the verification fails, the problem status changes to Pending. After you cancel the ignore status for a manually identified problem, the problem status changes to Pending. You can then perform other administration operations on the problem, such as initiating remediation or adding it to the whitelist.
Notify Problem Owner	You can notify the problem owner by internal message, phone, text message, email, or DingTalk.
Change Problem Owner	Transfers ownership of the current problem to another user. In the Change Problem Owner dialog box, select a recipient and click OK. We recommend syncing with the new owner after the transfer.
Mark as Resolved	Only manually entered problems in the Pending state can be marked as resolved. If remediation is required, you can reject the resolved status.
Reject and Reprocess	Only manually entered problems in the Completed state can be rejected for reprocessing. After being rejected, the problem status changes to Pending. Notify the problem owner through an internal message so they can process it promptly. Note Only super administrators, quality administrators, problem owners, and custom global roles with administration workbench flow management permissions can reject and reprocess problems.
Modify Problem Type	Classify problems by type for easier filtering and analysis. System-detected problems are automatically assigned a system problem type based on the quality rule type, which cannot be modified. Manually configured problem types can be added or deleted. If a suitable problem type is not available, you can create one. For more information, see Custom problem types.
Delete Problem	Deleting a problem also removes it from the remediation flow. This action cannot be undone.
Add to Whitelist	Example scenario: a table is in the testing phase and its data is unstable. You can add it to a whitelist until the data stabilizes. For more information about joining a whitelist, see Join a whitelist. For more information about managing whitelists, see View and manage administration whitelists. Note Only system-detected problems that have not been ignored can be added to the whitelist. Other problems are automatically skipped. After a problem is added to the whitelist, the system automatically ignores any errors from the current verification. The corresponding quality rule continues to run and verify, generate quality reports, send alerts, and block downstream tasks. While the whitelist is active, new quality problems that fall under the same rule scope are not reported. After the whitelist expires or is deleted, historical problems are not retroactively reported, but new problems that occur afterward are reported.
Re-verify	Only system-detected problems in the Pending or Ignored state can be re-verified. If a data quality problem has been resolved, re-run the verification rule. The problem status changes to Completed if verification passes, or to Pending if it fails.
View Operation Log	Shows operation records for the current problem, including the problem object name, operation type, problem status, operation description, operator, and operation time.
Associate Knowledge Base Document	Only system-detected problems can be associated with knowledge. After a rule is associated with knowledge, you can view the associated knowledge in quality rules and the problem checklist. You can associate a knowledge base with problem objects in batches. To create a knowledge base, see Create a knowledge base.

Batch Operations

Action	Description
Ignore This Abnormal Result	Changes the status of pending problems to ignored. We recommend ignoring unimportant problems. Example scenarios: a data source connection was temporarily interrupted but is now stable, or an abnormal result occurred in a historical task no longer used in current data processing.
Cancel Ignore	Only problems in the Ignored state can have their ignore status canceled. After you cancel the ignore status, manually entered problems change to the Pending state. For system-detected problems, the status is determined by the latest rule verification result. If the verification passes, the problem status changes to Completed. If the verification fails, the problem status changes to Pending.
Add to Whitelist	Scenario: A table is in the testing phase and its data is unstable. You can add the table to a whitelist to prevent monitoring until the data stabilizes. For more information, see Add to Whitelist. To manage the whitelist, see View and Manage Administration Whitelists. When adding problem objects to the whitelist in batches, the supported whitelist scopes for Dataphin tables, global data tables, metrics, data sources, and real-time metatables are Selected problems and Selected monitored objects. Selected problems: Within the validity period, when new quality verification failures occur for any of the selected pending or validating problems on the current page, their status is automatically changed to ignored. Selected monitored objects: When new quality verification failures occur for the problems selected on the current page and for all other pending or validating problems under the same monitored object, their status is automatically changed to ignored. Note Only system-detected, non-ignored problems can be added to the whitelist. Other problems are automatically skipped. After a problem is added to the whitelist, the system automatically ignores any errors from the current verification. The corresponding quality rule continues to run and verify, generate quality reports, send alerts, and block downstream tasks. While the whitelist is active, new quality problems that fall under the same rule scope are not reported. After the whitelist expires or is deleted, historical problems are not retroactively reported, but new problems that occur afterward are reported.
Initiate Remediation	After initiating remediation, you can track the entire remediation flow. The batch remediation flow works the same as a single operation. For more information, see Initiate Remediation. For information about how to manage the remediation flow, see View and manage remediation flows. When initiating remediation for problem objects in batches, the supported remediation scopes for Dataphin tables, global data tables, metrics, data sources, and real-time metatables are Selected problems, Selected problems and related rule problems, and Selected monitored objects. The supported remediation scopes for tags are Selected problems and All objects for selected tags. Selected problems: Remediation is performed only for the problems selected on the current page. Selected problems and related rule problems: Remediation is performed for the problems selected on the current page and for other pending problems that belong to the same rule as the selected system-detected problems. Selected monitored objects: Remediation is performed for the problems selected on the current page and for all other pending problems under the same monitored object, including both manually entered and automatically detected problems. All objects for selected tags: Remediation is performed for the problems selected on the current page and for all other pending problems under the same tag.
Re-verify	Only system-detected problems in the Pending or Ignored state can be re-verified. If a data quality problem has been resolved, re-run the verification rule for the selected problems. The problem status changes to Completed if verification passes, or to Pending if it fails.
Mark as Resolved	Only manually entered problems in the Pending state can be marked as resolved. If remediation is required, you can reject the resolved status.
Reject and Reprocess	Only manually entered problems in the Completed state can be rejected for reprocessing. After being rejected, the problem status changes to Pending. Note Only super administrators, quality administrators, problem owners, and custom global roles with administration workbench flow management permissions can reject and reprocess problems.
Modify Problem Type	Append or modify problem types. This only affects custom problem types, not system problem types automatically generated based on the rule type. Append: Adds the configured problem type to the existing problem types. Modify: Overwrites the existing custom types with the configured problem type.
Export Problems	Exports the selected problems. In the dialog box, select the content to export and click OK.
Change Problem Owner	Transfers ownership of the current problem to another user. In the Change Problem Owner dialog box, select a recipient and click OK. We recommend syncing with the new owner after the transfer.
Associate Knowledge Base Document	Only system-detected problems can be associated with knowledge. After a rule is associated with knowledge, you can view the associated knowledge in quality rules and the problem checklist. You can associate a knowledge base with problem objects in batches. To create a knowledge base, see Create a knowledge base.

Add to Whitelist

Add entries to the whitelist individually or in batches.

On the Problem Checklist page, select the target problem, click the icon in the Actions column, and select Add To Whitelist.

In the Add to Whitelist dialog box, configure the parameters.

Parameter	Description
Whitelist Scope	Only system-detected, non-ignored problems can be added to the whitelist. When selecting a single problem object, you can add the Current rule or Current monitored object to the whitelist. Current rule: Within the validity period, when new quality verification failures occur for the problem itself or for any other pending or validating problems under the same quality rule, their status is automatically changed to ignored. Current monitored object: Within the validity period, when new quality verification failures occur for any pending or validating problems under the same monitored object as the current problem, their status is automatically changed to ignored.
Validity Period	You can select 7 days, 30 days, 90 days, 180 days, Permanent, or Custom. The application day is not included in the count. Note After the whitelist expires, historical problems are not retroactively reported, but new problems that occur afterward are reported.
Operation Description	Required. Enter the reason for adding the problem to the whitelist. The description cannot exceed 128 characters.

Click OK to add the problem to the whitelist.

Initiate Remediation

Initiate remediation for problems individually or in batches.

On the Problem Checklist page, select the target problem and click the icon in the Actions column to open the Initiate Remediation dialog box.

In the Initiate Remediation dialog box, configure the parameters.

Parameter	Description
Remediation Scope	For system-detected problem objects, you can initiate remediation for the Current problem, Current rule, or Current monitored object. For manually entered problem objects, you can only initiate remediation for the Current problem. Current problem: Includes only the problem from the current verification record. It does not include other pending problems that belong to the same rule as the current problem object. Current rule: Includes the problem from the current verification record and all other pending, system-detected problems generated under the same quality rule. This may involve multiple check scopes. Current monitored object: Includes the problem from the current verification record and all other pending, system-detected problems under the same monitored object. This may involve multiple problem objects and check scopes. Note If a selected problem is already in a remediation flow, it remains in the original flow. A new remediation flow is not initiated.
Initiation Method	You can create a new flow or associate with an existing flow. Create New Flow: Enter a custom name for the remediation flow. The name cannot exceed 128 characters. Associate with Existing Flow: You can associate with an existing flow that is in the Remediating or Re-remediating state for the current object type. Only super administrators, quality administrators, problem owners, and custom global roles with administration workbench flow management permissions can select from all flows.
Remediate By	The remediator is the owner of the remediation process within the flow. We recommend that you select the quality owner of the problem object. This allows them to edit system-detected problems or re-initiate verification within the remediation flow. Otherwise, the problem status may not be updated. If the initiation method is Associate with Existing Flow, this field defaults to the remediator of the selected flow and cannot be modified.
Validated By	The validator is the final owner of the remediation result in the flow. If the initiation method is Associate with Existing Flow, this field defaults to the validator of the selected flow and cannot be modified.
Priority	There are three levels: High, Medium, and Low. If the initiation method is to associate with an existing flow, the priority defaults to that of the selected flow and cannot be modified.
Operation Description	Enter the reason for initiating remediation. The description cannot exceed 128 characters.

Click OK to initiate the remediation.

Export the problem checklist

Note

You can export only problem checklists that you have permission to view.
A single tenant supports up to 5 concurrent export tasks.
If a problem checklist contains a large amount of data, the file generation may take some time. You can view historical export records on the batch export panel. For more information, see Problem export records.

On the Problem Checklist page, click the Export Problems button in the upper-right corner. You can choose to Export all problems or Export based on search and filter criteria.
- Export all problems: Exports all problems of the specified object type from the current perspective.
- Export based on search and filter criteria: Exports all problems from the current perspective that match the keyword search and filter criteria. For example, you can filter and export all problems with the Pending status.
In the dialog box, select the content to export.
Click OK to export the problems.

Problem export records

Note

The last 50 export operation records are displayed.
You can download successfully exported files within 7 days.

On the Problem Checklist page, click the Export Problems button in the upper-right corner and select View Batch Export Records.
In the Batch Export Records panel, you can check the status of the problem export. The details include the ordinal number, object type, submission time, and operation status for each problem. You can also click the icon in the Actions column to download the problems to your local computer for statistical analysis and archiving.