Dataphin:Overview of data quality

Prerequisites

The data quality value-added service must be purchased and the data quality module enabled for the current tenant.

Background information

As organizations scale their data infrastructure, pipelines grow more complex and errors become harder to catch manually. Dataphin data quality standardizes raw business data across five dimensions—timeliness, accuracy, completeness, consistency, and validity—so your downstream analyses and decisions are built on reliable data.

Data quality process guide

The data quality process guide walks you through: (optional) configuring rule templates -> introducing monitored objects -> configuring quality rules -> validating rules -> viewing validation records and viewing quality reports -> performing quality rectification.

Scenarios for quality rules

Quality rules enforce data integrity throughout the pipeline. When a rule detects an anomaly during validation, whether it blocks downstream processing depends on how you configured the rule—as a hard rule or a soft rule.

• A hard rule triggers an alert and blocks downstream task nodes when the quality rule validation result is abnormal. Use hard rules to prevent bad data from propagating to production tables or downstream consumers.

• A soft rule triggers an alert but does not block downstream task nodes when the quality rule validation result is abnormal. Use soft rules to monitor data health without interrupting the pipeline—useful during gradual rollouts or when data issues are expected and tracked separately.

Feature overview

• Data quality supports rule validation and rectification across five types of monitored objects: Dataphin tables, global tables, metrics, data sources, and real-time meta tables.

  • Dataphin tables support rule validation and rectification across physical tables, logical fact tables, logical dimension tables, and logical aggregate tables.

  • Global tables support rule validation and rectification across external data sources, including MaxCompute, Hive, MySQL, Oracle, Microsoft SQL Server, PostgreSQL, SAP HANA, AnalyticDB for PostgreSQL, ClickHouse, IBM DB2, DM, and Hologres.

  • Metrics support monitoring, anomaly alerts, and corrective actions for field group quantity, duplicate field values, field stability, and volatility.

  • Data sources support monitoring, anomaly alerts, and corrective actions for connectivity changes and table structure changes.

  • Real-time meta tables support statistical value detection, real-time versus offline comparison, multi-link real-time comparison, anomaly alerts, and corrective actions.

• Data quality delivers an end-to-end solution covering rule validation, monitoring, intelligent alerting, report generation, and rectification across all monitored object types—keeping data reliable throughout the production and consumption lifecycle.

Data quality encompasses Quality Overview, Quality Monitoring, and Quality Administration:

• Quality Overview shows the number of validated tables, tables with abnormal results, and other key metrics—making it easy to spot and address validation issues at a glance.

• Quality Monitoring provides access to the quality rule list, rule configuration, validation records, and quality reports.

• Quality Administration is where you review validation errors and take action—initiating rectification, dismissing issues, or sending notifications. This closes the Plan-Do-Check-Act (PDCA) loop and drives continuous improvement in data quality.