Data Transmission Service:Configure a data verification task

Usage notes

• Incremental data verification is not supported for database instances that are deployed in a classic network.

• Data verification is not supported for DTS instances for which the multi-table merging feature is enabled.

• Data verification is not supported for instances whose Status is Completed.

• If a table to be verified contains neither a primary key nor a unique index and the table contains more than 10,000 rows of data, DTS does not perform data verification on the table.

• If the data to be verified has neither a PRIMARY KEY constraint nor a UNIQUE constraint, DTS does not perform incremental data verification.

• If you restart a synchronization or migration instance, such as by pausing and then starting it, the full verification task in the instance restarts.

• DTS does not perform data verification on a destination table if it is empty.

• Before you perform a verification task, evaluate the performance of the source and destination databases. We recommend that you perform data verification during off-peak hours, for example, when the CPU load of the source and destination databases is below 30%. During data verification, DTS consumes read resources from the source and destination databases, which may increase the database load. If the database performance is poor, the specifications are low, or the business volume is high, the load on the database may increase and even cause the database services to become unavailable.

• If you configure a data verification task in a data synchronization or migration instance, the valid values for Data Verification Mode are determined by the values that you select for Synchronization Types or Migration Types.

  For example, if you select only Incremental Data Migration for Migration Types, Incremental Data Verification is the only valid value for Data Verification Mode.

• An incremental verification task verifies only the data that is changed during an incremental synchronization or migration task. It does not verify data that is manually changed in the destination database.

• Incremental verification is not supported for verification tasks that you configure on the Data Verification page.

• When you verify a small amount of data using the Full field validation by row sampling method, the sampling percentage that you set does not take effect. The default value is 100%.

• If you select Full field validation by row sampling when you configure data verification for a MongoDB instance, the sampling percentage is fixed to 100%.

• If you select Full field validation by row sampling for Full Data Verification, we recommend that you do not use the extract, transform, and load (ETL) feature. Otherwise, the data modified using the ETL feature is detected as inconsistent data.

• Data verification is not supported for materialized views.

• During a full verification task, if a single table contains more than 100,000 rows of inconsistent data, DTS stops the verification of that table. Reduce the amount of inconsistent data and then verify the table again.

• If an object to be verified contains hidden columns, the schema verification result may be affected.

Supported regions

For more information, see Supported regions.

Supported scenarios

Configure a data verification task on the Data Verification page

1. Go to the verification task list page. You can use one of the following three methods.

   Use the DTS console (quick start)

   1. Go to the DTS Verification Tasks page.

   2. Optional: In the upper-left corner of the page, select the region where the verification instance resides.

   Use the DTS console (standard)

   1. Log on to the Data Transmission Service (DTS) console.

   2. In the navigation pane on the left, click Data Verification.

   3. Optional: In the upper-left corner of the page, select the region where the verification instance resides.

   Use the DMS console

   Note

   The actual steps that you need to perform may vary based on the mode and layout of the DMS console. For more information, see Simplified mode console and Customize the layout and style of the DMS interface.

   1. Log on to DMS Data Management Service.

   2. In the top menu bar, select Data + AI > Data Transmission Service (DTS) > Data Verification.

   3. Optional: To the right of Verification Task, select the region where the verification instance is located.

2. Go to the configuration page of a data verification task in one of the following ways:

   • If you have purchased a data verification task, click Configure Task in the Actions column of the target task.

     For more information about how to purchase a data verification instance, see Purchase a data verification instance.

   • If you have not purchased a data verification task, click Create Task.

3. Configure the source and destination databases. For more information, see Overview of data synchronization solutions or Overview of migration solutions.

   Note

   • You cannot modify the Instance Region for tasks that you purchase before you configure them.

   • If the source database is a MongoDB sharded cluster, you do not need to prepare the Shard account or the Shard password.

4. In the lower part of the page, click Test Connectivity and Proceed.

   Note

   • Make sure that the CIDR blocks of DTS servers can be automatically or manually added to the security settings of the source and destination databases to allow access from DTS servers. For more information, see Add DTS server IP addresses to a whitelist.

   • If the source or destination database is a self-managed database and its Access Method is not set to Alibaba Cloud Instance, click Test Connectivity in the CIDR Blocks of DTS Servers dialog box.

5. Configure Data Verification Mode.

   Select one or more data verification modes as needed. If you select Full Data Verification, you must also configure the parameters that are described in the following table.

   Note

   • Full Data Verification: Verifies the full data of the specified objects.

   • Schema Verification: Verifies the database and table schemas of the specified objects.

   • Incremental verification is not supported for verification tasks that you configure on the Data Verification page.

   Parameter	Note
   Full Data Verification	Full field validation by row sampling: Verifies all fields for a specified percentage of data rows. Set the sampling percentage to an integer from 10 to 100. Verify based on the number of table rows: Verifies the row count of the full dataset. The actual data content is not verified. Note The Verify based on the number of table rows mode is free of charge. The Full field validation by row sampling mode is charged based on the amount of data that is verified.
   Full Data Verification Time Rule	Currently, only Start Now is supported.
   Timeout Settings for Full Data Verification	No: The full verification task is not forcibly stopped on timeout. Yes: Set a timeout period for the full verification task. The timer starts after the task begins. If the task does not complete within the specified time, it is forcibly stopped. The value must be an integer from 1 to 72.
   Full calibration reference	Default: Uses the combined data from the source and destination databases as the benchmark to verify data consistency. Source Database: Uses the source database as the benchmark to verify data consistency. Extra data in the destination database is not checked. Destination Database: Uses the destination database as the benchmark to verify data consistency. Extra data in the source database is not checked.
   Maximum number of rows of data read per second by full verification (RPS)	Full data verification consumes database read resources. To reduce the load on the database, you can configure throttling for the full verification task as needed. Throttling limits the number of data rows and the amount of data read per second. Note A parameter value of 0 indicates no limit. If both Maximum number of rows of data read per second by full verification (RPS) and The maximum number of bytes read per second by full verification Byte/s. are set to 0, throttling is disabled.
   The maximum number of bytes read per second by full verification Byte/s.

6. Specify Verification Objects.

   1. Select the objects to be verified in the source database.

      In the Source Objects section, select the objects that you want to verify and click to move them to the Selected Objects section.

      Note

      You can set Table Name and Column Name for a destination table only if you select tables as the objects to be verified.

   2. Set the objects to be verified in the destination database.

      1. In the Selected Objects section, right-click the schema that you moved.

      2. In the Edit Schema dialog box, set Schema Name for the destination database, and click OK.

      3. In the Selected Objects section, right-click a table that you moved from the Source Objects section.

      4. In the Edit Table dialog box, set the Table Name for the destination table.

      5. Optional. In the Column Name column of the Column section, set the column name of the destination table.

         Note

         You can also deselect Synchronize All Tables and manually select the columns to verify.

      6. Click OK.

   3. Optional. Configure verification alerts.

      If you select Full Data Verification for Data Verification Mode, you can also configure Full Data Verification Alert as needed.

      Note

      After the DTS instance starts running, you can also set or modify data verification alerts.

      Parameter	Note
      Full Data Verification Alert	No: Disables alerting. Yes: Enables alerting. Select and configure the alert rules. The alert rules are as follows: An alert is triggered if the full data verification task fails. An alert is triggered if the amount of inconsistent data from the full data verification task is greater than or equal to the specified threshold.

      Important

      If you have set verification alerts and want to receive alert notifications when a verification task triggers an alert, you must also configure alert message subscriptions on the Cloud Monitor platform. For more information, see Set alert rules for DTS tasks on the Cloud Monitor platform.

7. Click Next: Advanced Settings to configure advanced settings.

   Configuration	Description
   Dedicated Cluster for Task Scheduling	By default, DTS schedules the task to a shared cluster. No selection is required. You can also purchase a dedicated cluster to run DTS tasks. For more information, see What is a DTS dedicated cluster.
   Retry Time for Failed Connections	The retry time range for failed connections. If the source or destination database fails to be connected after the data synchronization task is started, DTS immediately retries a connection within the time range. Valid values: 10 to 1440. Unit: minutes. Default value: 720. We recommend that you set this parameter to a value greater than 30. If DTS reconnects to the source and destination databases within the specified time range, DTS resumes the data synchronization task. Otherwise, the data synchronization task fails. Note If you specify different retry time ranges for multiple data synchronization tasks that have the same source or destination database, the shortest retry time range takes precedence. When DTS retries a connection, you are charged for the DTS instance. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at your earliest opportunity after the source and destination instances are released.
   Retry Time for Other Issues	The retry time range for other issues. For example, if the DDL or DML operations fail to be performed after the data synchronization task is started, DTS immediately retries the operations within the time range. Valid values: 1 to 1440. Unit: minutes. Default value: 10. We recommend that you set this parameter to a value greater than 10. If the failed operations are successfully performed within the specified time range, DTS resumes the data synchronization task. Otherwise, the data synchronization task fails. Important The value of the Retry Time for Other Issues parameter must be smaller than the value of the Retry Time for Failed Connections parameter.
   Environment Tag	Optional. As needed, select an environment tag to identify the instance.

8. Save the task settings and run a precheck.

   • To view the parameters to be specified when you call the relevant API operation to configure the DTS task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

   • If you do not need to view or have viewed the parameters, click Next: Save Task Settings and Precheck in the lower part of the page.

   Note

   • Before the verification task starts, a precheck is performed. The verification task can start only after it passes the precheck.

   • If the task fails to pass the precheck, click View Details next to each failed item. After you analyze the causes based on the check results, troubleshoot the issues. Then, run a precheck again.

   • If an alert is triggered for an item during the precheck:

     • If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.

     • If the alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur, and your business may be exposed to potential risks.

9. View the progress of the data verification task.

   • If you have a purchased verification task (purchased before configuration), when the Success Rate is 100%, click Back.

   • If you have configured a data verification task but have not purchased it, when the Success Rate is 100%, click Next: Purchase Instance, read and select Data Transmission Service (Pay-as-you-go) Service Terms, click Buy and Start, and then click OK.

Configure a data verification task in a DTS synchronization or migration instance

1. Go to the Data Verification page.

   • For information about how to configure a newly created DTS instance, see the topics listed in Overview of data synchronization solutions or Overview of migration solutions.

   • For an existing DTS instance:

     1. Go to the synchronization or migration task list page.

     2. Click the target instance ID.

     3. On the Basic Information or Task Management page, click Create Verification Task.

2. Configure Data Verification Mode.

   Note

   • Full Data Verification: Verifies the data in the full data task. If the instance does not include an incremental task, verification starts after the instance is created and the full data is completely written. If the instance includes an incremental task, verification starts after the instance is created and the incremental data is written for the first time with no latency.

   • Incremental Data Verification: Verifies data that is incrementally synchronized or migrated. Data verification starts after the instance is created and incremental data is written to the destination database with no latency for the first time.

   • Schema Verification: Verifies the schemas of the objects that require verification. If the instance does not include an incremental task, verification starts after the instance is created and the table schema and full data are written. If the instance includes an incremental task, verification starts after the instance is created and incremental data is written for the first time with no latency.

   Full verification

   If you select Full Data Verification, you also need to set the parameters described in the following table.

   Parameter	Note
   Full Data Verification	Full field validation by row sampling: Verifies all fields for a specified percentage of data rows. Set the sampling percentage to an integer from 10 to 100. Verify based on the number of table rows: Verifies the row count of the full dataset. The actual data content is not verified. Note The Verify based on the number of table rows mode is free of charge. The Full field validation by row sampling mode is charged based on the amount of data that is verified.
   Full Data Verification Time Rule	Currently, only Start Now is supported.
   Timeout Settings for Full Data Verification	No: The full verification task is not forcibly stopped on timeout. Yes: Set a timeout period for the full verification task. The timer starts after the task begins. If the task does not complete within the specified time, it is forcibly stopped. The value must be an integer from 1 to 72.
   Full calibration reference	Default: Uses the combined data from the source and destination databases as the benchmark to verify data consistency. Source Database: Uses the source database as the benchmark to verify data consistency. Extra data in the destination database is not checked. Destination Database: Uses the destination database as the benchmark to verify data consistency. Extra data in the source database is not checked.
   Maximum number of rows of data read per second by full verification (RPS)	Full data verification consumes database read resources. To reduce the load on the database, you can configure throttling for the full verification task as needed. Throttling limits the number of data rows and the amount of data read per second. Note A parameter value of 0 indicates no limit. If both Maximum number of rows of data read per second by full verification (RPS) and The maximum number of bytes read per second by full verification Byte/s. are set to 0, throttling is disabled.
   The maximum number of bytes read per second by full verification Byte/s.

   Incremental verification

   If you select Incremental Data Verification, you also need to set the parameters described in the following table.

   Parameter	Note
   Incremental Verification Benchmark	You can select the Data Manipulation Language (DML) operations to verify as needed.

3. Specify Verification Objects.

   In the Selected Objects section, you can select the objects that do not require data verification and click to remove them.

   Note

   By default, the objects to be synchronized or migrated are added to the Selected Objects section.

4. Configure alerts for data verification.

   Configure the following parameters as needed.

   Note

   After the DTS instance starts running, you can also set or modify data verification alerts.

   Parameter	Note
   Full Data Verification Alert	No: Disables alerting. Yes: Enables alerting. Select and configure the alert rules. The alert rules are as follows: An alert is triggered if the full data verification task fails. An alert is triggered if the amount of inconsistent data from the full data verification task is greater than or equal to the specified threshold.
   Incremental Data Verification Alert	No: Disables alerting. Yes: Set up alerts by selecting and configuring alert rules. The available alert rules are as follows: Triggers an alert if the incremental verification task fails. Triggers an alert if the number of inconsistent records meets or exceeds a threshold for a specified number of consecutive statistical periods. Configure the number of periods, the statistical period, and the threshold for inconsistent records. Triggers an alert if data latency meets or exceeds a threshold for a specified number of consecutive statistical periods. Configure the number of periods, the statistical period, and the latency threshold.

   Important

   If you have set verification alerts and want to receive alert notifications when a verification task triggers an alert, you must also configure alert message subscriptions on the Cloud Monitor platform. For more information, see Set alert rules for DTS tasks on the Cloud Monitor platform.

5. Follow the prompts and refer to the relevant configuration documents to complete the subsequent configuration.

Modify an alert rule

1. Go to the Data Verification page. You can use one of the following three methods.

   Note

   If you configured the data verification task in a DTS synchronization or migration instance, you can also go to the corresponding synchronization or migration task list page.

   Use the DTS console (quick start)

   1. Go to the DTS Verification Tasks page.

   2. Optional: In the upper-left corner of the page, select the region where the verification instance resides.

   Use the DTS console (standard)

   1. Log on to the Data Transmission Service (DTS) console.

   2. In the navigation pane on the left, click Data Verification.

   3. Optional: In the upper-left corner of the page, select the region where the verification instance resides.

   Use the DMS console

   Note

   The actual steps that you need to perform may vary based on the mode and layout of the DMS console. For more information, see Simplified mode console and Customize the layout and style of the DMS interface.

   1. Log on to DMS Data Management Service.

   2. In the top menu bar, select Data + AI > Data Transmission Service (DTS) > Data Verification.

   3. Optional: To the right of Verification Task, select the region where the verification instance is located.

2. In the Actions column for the target verification task, select > Alert Settings.

   Note

   You can also click the target instance ID, and then click Alert Settings in the navigation pane on the left.

3. In the Actions column of the target metric, click Edit.

   • Full data verification

     • Full Data Verification Status

     • Full Data Verification Inconsistency

   • Incremental data verification

     • Incremental Data Verification Status

     • Incremental Data Verification Latency

     • Incremental Data Verification Inconsistency

4. After you make the required changes, click OK.

Related API operations

FAQ

• Are database, table, and column names that are modified using the name mapping feature verified as inconsistent data?

  The data verification feature is compatible with the name mapping feature for databases and tables. If you use this feature to modify database, table, and column names, only the modified column names are detected as inconsistent data. The database and table names are not.

• What is the result of a schema verification task if the table contains hidden columns?

  A schema verification task verifies the position of columns. If a table to be verified contains hidden columns, the positions of other columns may be affected. If the position of a column to be verified in the source database is different from that in the destination database, DTS reports a schema inconsistency.

• Why can I not select a verification mode in a DTS instance?

  The current instance may not support the corresponding data verification mode, or the corresponding Synchronization Types or Migration Types was not selected in the Configure Objects step.

• Is data that is filtered by the filter condition feature verified as inconsistent data?

  • Full data verification: No. Full data verification supports the Filter Conditions feature.

  • Incremental data verification: No. Incremental data verification supports the Filter Conditions feature. (Incremental tasks for a source database whose Database Type is MongoDB do not support the Filter Conditions feature.)