This topic describes how to configure a data verification task in Data Transmission Service (DTS). The data verification feature helps you detect inconsistent data between the source and destination databases at the earliest opportunity.
Prerequisites
Full Data Synchronization or Full Data Migration is selected as Task Stages if you need to configure full data verification for a DTS task.
Incremental Data Synchronization or Incremental Data Migration is selected as Task Stages if you need to configure incremental data verification for a DTS task.
Usage notes
Incremental data verification is not supported for database instances that are deployed on the classic network.
Data verification is not supported for DTS tasks for which the multi-table merging feature is enabled.
Data verification is not supported for DTS tasks that are in the Completed state.
If a table to be verified contains neither a primary key nor a unique index and the table contains more than 10,000 rows of data, DTS does not perform data verification on the table.
If the data to be verified has neither PRIMARY KEY nor UNIQUE constraints, DTS does not perform incremental data verification.
If you pause and then restart a data synchronization or migration task, full data verification configured for the DTS task is also restarted to verify data again.
Before you perform data verification, you must evaluate the performance of the source and destination databases. We recommend that you perform data verification during off-peak hours. For example, the CPU load of the source and destination databases is less than 30%. During data verification, DTS uses the read resources of the source and destination databases. This may increase the loads of the database servers. If you perform data verification when the database performance is poor, the database specifications are low, or the business volume is large, high pressure may be caused on the databases. The database services may become unavailable.
DTS performs incremental data verification only on the data that is changed during incremental data synchronization or migration, but not on the data that is manually changed in the destination database.
Supported regions
Data verification is supported in all regions and cross-region scenarios. For more information, see Supported regions.
Supported scenarios
Data verification is supported in two-way synchronization scenarios between the following source and destination databases.
The following source and destination databases support Alibaba Cloud database instances and self-managed database instances.
Data verification mode | Source database type | Destination database type |
Full data verification | MySQL | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, PostgreSQL, AnalyticDB for PostgreSQL, and Oracle |
PolarDB for MySQL | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, PostgreSQL, AnalyticDB for PostgreSQL, PolarDB for PostgreSQL, and Oracle | |
PostgreSQL | MySQL, PostgreSQL, AnalyticDB for PostgreSQL, PolarDB for PostgreSQL, and PolarDB for Oracle | |
PolarDB for PostgreSQL | PostgreSQL, AnalyticDB for PostgreSQL, PolarDB for PostgreSQL, and Oracle | |
Standalone MongoDB databases | Standalone MongoDB databases | |
MongoDB databases that use the replica set architecture | MongoDB databases that use the replica set architecture | |
MongoDB databases that use the sharded cluster architecture | MongoDB databases that use the sharded cluster architecture | |
Tair and Redis databases that use the master-replica architecture | Tair and Redis databases that use the master-replica architecture | |
Tair and Redis databases that use the cluster architecture | Tair and Redis databases that use the cluster architecture | |
SQL Server | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, SQL Server, and AnalyticDB for PostgreSQL | |
Oracle | AnalyticDB for MySQL, AnalyticDB for PostgreSQL, MySQL, PolarDB for MySQL, PolarDB for PostgreSQL, PolarDB for Oracle, PostgreSQL, and Oracle | |
PolarDB for Oracle | PolarDB for Oracle, AnalyticDB for MySQL, AnalyticDB for PostgreSQL, and Oracle | |
Db2 for LUW | MySQL, PolarDB for MySQL, AnalyticDB for MySQL, and AnalyticDB for PostgreSQL | |
Db2 for i (AS/400) | MySQL | |
Incremental data verification | MySQL | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, PostgreSQL, AnalyticDB for PostgreSQL, and PolarDB for PostgreSQL |
PolarDB for MySQL cluster | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, and AnalyticDB for PostgreSQL | |
PostgreSQL | MySQL, PostgreSQL, AnalyticDB for PostgreSQL, and PolarDB for PostgreSQL | |
PolarDB for PostgreSQL | PostgreSQL, AnalyticDB for PostgreSQL, and PolarDB for PostgreSQL | |
SQL Server | MySQL, AnalyticDB for MySQL, PolarDB for MySQL, SQL Server, and AnalyticDB for PostgreSQL | |
Oracle | AnalyticDB for MySQL, MySQL, PolarDB for MySQL, and PolarDB for Oracle | |
Schema verification | MySQL | MySQL and PolarDB for MySQL |
PolarDB for MySQL | MySQL and PolarDB for MySQL | |
PostgreSQL | PostgreSQL and PolarDB for PostgreSQL | |
PolarDB for PostgreSQL | PostgreSQL and PolarDB for PostgreSQL |
Configure the data verification feature for a data synchronization or migration task in DTS
Go to the Advanced Settings step.
For information about how to go to the Advanced Settings step when you create a DTS task, see the topics listed in Overview of data synchronization scenarios or Overview of data migration scenarios.
For an existing DTS task, perform the following steps:
Go to the Data Synchronization Tasks or Data Migration Tasks page in the DTS console.
Find the DTS task that you want to manage and click the task ID.
On the Basic Information or Task Management page, click Create Verification Task.
Select data verification modes.
Select one or more data verification modes based on your business requirements. If you select Full Data Verification, you must configure the parameters that are described in the following table.
NoteFull Data Verification: verifies data in the full data synchronization or migration stage of the DTS task. If the task contains no incremental data synchronization or migration stage, data verification starts after the task is created and full data is written to the destination database. If the task contains the incremental data synchronization or migration stage, data verification starts after the task is created and incremental data is written to the destination database without latency for the first time.
Incremental Data Verification: verifies data in the incremental data synchronization or migration stage of the DTS task. Data verification starts after the task is created and incremental data is written to the destination database without latency for the first time.
Schema Verification: verifies the schemas of objects in the data synchronization or migration task. If the task contains no incremental data synchronization or migration stage, data verification starts after the task is created and schemas and full data are written to the destination database. If the task contains the incremental data synchronization or migration stage, data verification starts after the task is created and incremental data is written to the destination database without latency for the first time.
Full data verification and incremental data verification start after schema verification is complete.
Parameter
Description
Full Data Verification
Verify hash values based on the sampling ratio: verifies all fields of the data sampled based on the specified sampling percentage. You can specify an integer that ranges from 10 to 100 in the field as the sampling percentage.
Verify based on the number of table rows: verifies the number of table rows in the full data synchronization or migration stage of the DTS task. DTS does not verify the data content.
NoteIf the full data verification mode is Verify based on the number of table rows, you are not charged. If the full data verification mode is Verify hash values based on the sampling ratio, you are charged based on the actual amount of data to be verified.
Full Data Verification Time Rule
You can set this parameter only to Start Now.
Timeout Settings for Full Data Verification
No: does not specify a timeout period for full data verification. The system does not forcibly stop full data verification if the verification times out.
Yes: specifies a timeout period for full data verification. The system starts to count the duration when full data verification starts. If full data verification is not complete within the specified timeout period, the system forcibly stops the verification. The value must be an integer that ranges from 1 to 72.
Select Verification Benchmark
Default: checks the data consistency between the source and destination databases based on the source and destination databases.
Source Database: checks the data consistency between the source and destination databases based on the source database. Objects that exist in the destination database but do not exist in the source database are not checked.
Destination Database: checks the data consistency between the source and destination databases based on the destination database. Objects that exist in the source database but do not exist in the destination database are not checked.
The maximum number of request rows that are read per second.
During full data verification, DTS uses the read resources of the source and destination databases. You can enable throttling for full data verification based on your business requirements. To configure throttling, you must specify the maximum number of data rows and bytes that are read per second. This reduces the loads of the database servers.
NoteA value of 0 specifies that the full data verification speed is not limited. If you set the The maximum number of request rows that are read per second. and The maximum number of bytes that are read per second. Unit: Byte/s. parameters to 0, throttling is disabled.
The maximum number of bytes that are read per second. Unit: Byte/s.
Specify the objects to be verified.
You can select objects that do not require data verification in the Selected Objects section, and then click the icon to remove the objects.
NoteBy default, the objects to be synchronized or migrated are added to the Selected Objects section.
Configure verification alerts.
Configure the parameters for configuring verification alerts based on your business requirements. The following table describes the parameters.
Parameter
Description
Full Data Verification Alert
No: does not configure alerts.
Yes: configures alerts. You must also select and configure alert rules. The following information describes the alert rules:
An alert is triggered if full data verification fails.
An alert is triggered if the volume of inconsistent data detected by full data verification is larger than or equal to the specified threshold.
Incremental Data Verification Alert
No: does not configure alerts.
Yes: configures alerts. You must also select and configure alert rules. The following information describes the alert rules:
An alert is triggered if incremental data verification fails.
An alert is triggered if the volume of inconsistent data detected by incremental data verification within the specified consecutive periods is larger than or equal to the specified threshold. You can specify the number of consecutive periods, the statistical period, and the threshold of the inconsistent data volume.
An alert is triggered if the latency of data migration or synchronization detected by incremental data verification within the specified consecutive periods is greater than or equal to the specified threshold. You can specify the number of consecutive periods, the statistical period, and the latency of data migration or synchronization.
Complete the subsequent steps as prompted.
Related API operations
Operation | Description |
Configures a data verification task. |