Data migration is a common scenario in which data accuracy directly affects business operations. Therefore, ensuring data consistency before and after migration is critical. Cloud Migration Hub provides multiple verification methods for various data sources, including database and object storage services, to ensure data consistency between data sources and destinations. This topic describes how to use the data verification feature.
1. Use the data verification feature
The process of using the data verification feature includes the following steps:
Step 1: install a data verification agent. Cloud Migration Hub needs to access and compare data from both sources and destinations to perform data verification. This may involve sensitive data. To prevent sensitive data leaks, you must provide a server that is connected to both data sources and destinations to deploy a data verification agent. We recommend that you use an Alibaba Cloud Elastic Compute Service (ECS) instance.
Step 2: configure a data source and create a verification plan. In most scenarios, you need to verify only specific data within a data source. Therefore, you must specify a data scope and method for verification. Then, select an agent to perform the verification.
Step 3: start verification. After you create a verification task, you can send a verification command to the agent specified for the task. The agent automatically returns verification results after verification is complete. No additional operations are required. You only need to wait for completion.
Step 4: view results. After the agent returns results, you can view the results and details in the Cloud Migration Hub console.
These steps enable GUI-based data consistency verification between sources and destinations, ensuring smooth data cutover.
2. Feature overview
2.1 Scenarios
Cloud Migration Hub provides an efficient and lightweight data verification feature, specifically designed for data migration in database and object storage service scenarios. After migration is complete, the feature samples data to check data consistency between the source and destination, thereby ensuring business stability. During a system cutover window, the feature implements quick verification after system shutdown to provide real-time quality assurance for data migration.
Verification timing
To ensure verification accuracy, we recommend that you start verification when the following conditions are met:
Migration is complete on both the data source and destination.
Incremental synchronization or migration is complete.
Precautions
If you perform verification when data migration is still in progress, including full migration and incremental synchronization phases, the following issues may occur because data at the source and destination ends can undergo continuous changes:
Inaccurate verification results
Data consistency misjudgment
Migration process interference
Cloud Migration Hub adopts a lightweight data verification algorithm to maintain high verification accuracy while minimizing resource overhead, delivering reliable quality assurance for data migration in key business systems and helping enterprises achieve seamless and secure data migration.
2.2 Supported services and verification methods
Database services
Supported service | Count | TopN | SumX | CheckSum | ObjectStore |
MySQL | Obtains the number of rows in a table. | Obtains the first n rows in a table for comparison. | Obtains the sum of specified fields in a table. | Obtains the checksum of a table. | Not supported. |
Redis | Compares the value sizes of a specified key. | Compares the n value attributes of a specified key. | Compares the values of keys that are specified or sampled between source and destination ends. For example, you can specify 100 keys. | Not supported. | Not supported. |
MongoDB | Obtains the number of documents in a collection. | Obtains the first n documents in a collection for comparison. | Obtains the sum of specified fields in a collection. | Not supported. | Not supported. |
Elasticsearch | Obtains the number of documents in an index. | Obtains the first n documents in an index for comparison. | Not supported. | Not supported. | Not supported. |
For ordered data of Redis, the TopN method is used for verification. For unordered data of Redis, the Count method is used.
Object storage services
Supported service | ListObject | ContentCheck | Inventory |
Alibaba Cloud Object Storage Service (OSS) | Compares the names and sizes of all objects in a bucket. The sizes are measured in bytes. | Performs MD5 verification on each object in the report generated by the online migration service. | Compares objects based on the inventory lists that are generated. If inconsistent objects exist, the getObject operation is called for secondary verification. |
Huawei Cloud Object Storage Service (OBS) | Compares the names and sizes of all objects in a bucket. The sizes are measured in bytes. | Performs MD5 verification on each object in the report generated by the online migration service. | Compares objects based on the inventory lists that are generated. If inconsistent objects exist, the getObject operation is called for secondary verification. |
Amazon Simple Storage Service (Amazon S3) | Compares the names and sizes of all objects in a bucket. The sizes are measured in bytes. | Not supported. | Compares objects based on the inventory lists that are generated. If inconsistent objects exist, the getObject operation is called for secondary verification. |
Tencent Cloud Object Storage (COS) | Compares the names and sizes of all objects in a bucket. The sizes are measured in bytes. | Performs MD5 verification on each object in the report generated by the online migration service. | Compares objects based on the inventory lists that are generated. If inconsistent objects exist, the getObject operation is called for secondary verification. |
3. Procedure
3.1 Apply for and view a license
Log on to the Cloud Migration Hub console. In the left-side navigation pane, choose Migration Capability > Data Verification. If this is your first time to use the data verification feature, you can view a usage guide on the page that appears. The first step is to click Apply for a License. Then, a license is automatically generated. The license is used to protect data verification agents and prevent the agents from being abused. 
You can view the remaining validity period of the license. The default validity period is 90 days. You can also click View License to copy the content of the license.
3.2 Manage data verification agents
After you apply for a license, you must deploy a data verification agent. To deploy an agent, you must prepare one or more servers. You can determine the number of servers based on your business requirements.
Recommended operating system: CentOS or Alibaba Cloud Linux.
Network requirements: Your ECS instance is connected to the data source.
Cloud Migration Hub supports automatic and manual deployment of agents.
If you want to deploy an agent on an Alibaba Cloud ECS instance with Cloud Assistant installed, we recommend that you deploy the agent by using the automatic method. By default, Cloud Assistant is installed on Alibaba Cloud ECS instances.
If you want to deploy an agent on a third-party cloud server that is connected to an Alibaba Cloud virtual private cloud (VPC) or on an Alibaba Cloud ECS instance on which Cloud Assistant is not installed or fails to be installed due to specific reasons, we recommend that you deploy the agent by using the manual - private network method.
If you want to deploy an agent on a third-party cloud server that is not connected to Alibaba Cloud VPCs and uses a public IP address, we recommend that you deploy the agent by using the manual - public network method.
3.2.1 Automatic
This method is supported only in the China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), and China (Ulanqab) regions. If you want to use the method in other regions, contact online after-sales.
Take note of the following information:
You can use this method only if you use Alibaba Cloud ECS instances.
You must activate PrivateLink before you deploy an agent.
If you use PrivateLink to implement connectivity, a reverse endpoint is created within your account. You are charged for the endpoint. We recommend that you release it when it is no longer needed.
After the required ECS instance is prepared, click Agent Management.
In the panel that appears, click Agent Installation. 
In the dialog box that appears, set the Installation method parameter to Automated Deployment, enter an agent name, select the region in which you want to deploy the agent and the ECS instance on which you want to install the agent, and then click Installation.
During the deployment process, you can view the deployment progress of the agent.
After the agent is deployed, you can view the agent in the agent list.
Take note of the following information:
If you use PrivateLink to implement connectivity, a reverse endpoint is created within your account. You are charged for the endpoint. We recommend that you release it when it is no longer needed.
3.2.2 Manual
3.2.2.1 Manual - public network
Take note of the following information:
If you use this method, you can use servers on any platform or in data centers as jump servers. Make sure that Cloud Migration Hub can access your servers over the Internet.
After the required ECS instance is prepared, click Agent Management.
In the panel that appears, click Agent Installation.
In the dialog box that appears, set the Installation method parameter to Manual deployment, enter an agent name, set the LoadBalancerType parameter to Public network, enter an IP address and a port number, and then click Next step.
In the Installing the Agent step, an installation command is displayed. Click Copy to copy the command and then run the command on the ECS instance to install the agent.
After the agent is installed, click Next step. In the Get through the network step, click Connection to connect to the agent.
You can view the connection progress online.
After the agent is deployed, you can view the agent in the agent list.
3.2.2.2 Manual - private network
This method is supported only in the China (Hangzhou), China (Shanghai), China (Zhangjiakou), China (Shenzhen), and China (Ulanqab) regions. If you want to use the method in other regions, contact online after-sales.
Take note of the following information:
If you use this method, you can use Alibaba Cloud ECS instances or third-party cloud servers as jump servers. Make sure that your servers are connected to Alibaba Cloud VPCs.
You must activate PrivateLink before you deploy an agent.
After the required ECS instance is prepared, click Agent Management.
In the panel that appears, click Agent Installation.
In the dialog box that appears, set the Installation method parameter to Manual deployment, enter an agent name, set the LoadBalancerType parameter to Private network, select a region and a VPC, enter an IP address and a port number, and then click Next step.
In the Installing the Agent step, an installation command is displayed. Click Copy to copy the command and then run the command on the ECS instance to install the agent.
After the agent is installed, click Next step. In the Get through the network step, click Connection to connect to the agent.
You can view the connection progress online.
After the agent is deployed, you can view the agent in the agent list.
Take note of the following information:
If you use PrivateLink to implement connectivity, a reverse endpoint is created within your account. You are charged for the endpoint. We recommend that you release it when it is no longer needed.
3.2.3 Agent management operations
3.2.3.1 Stop
If you deploy an agent by using the automatic method and the agent is in the Online state, you can click Stop to stop the agent.
After the agent is stopped, the agent stops running on the related ECS instance, and the status of the agent changes to Stop in the agent list. If a data verification task uses an agent in the Stop state, you cannot start the task.
3.2.3.2 Start
If an agent is in the Stop state, you can click Start to start the agent.
After the agent is started, the status of the agent changes to Online.
3.2.3.3 Delete
If you no longer require an agent, you can delete the agent in the agent list.
3.2.3.4 Establish a new connection and copy a start command
If an agent is manually deployed and is disconnected, the status of the agent changes to Offline. You can click Copy Start Command and run the command on the related ECS instance. Then, click Reconnect to establish a new connection between Cloud Migration Hub and the agent.
After the connection is established, the status of the agent changes to Online.
3.3. Create a verification plan
3.3.1 Create a verification plan
OSS is used as an example to illustrate how to create a verification plan. If you want to verify the data of other services, follow the on-screen instructions for different verification methods in the Cloud Migration Hub console.
After your agent is started and the network is connected, click Create Verification Plan. In the panel that appears, configure the parameters.
Configure the Select Data Source parameter and the related parameters. Then, click Test data connection. After the test is successful, click Confirm to start data parsing.
The preceding figure shows only the configuration of a data source. If you want to verify data between the source and destination ends, you must also configure a data destination.
Wait until the data parsing is complete. Then, the verification plan is created. 
3.3.2 Create a data verification task
Click the name of the data verification plan to go to the details page of the plan.
On the page that appears, you can view information about your agent. A data verification plan supports only one agent. Click New Task to create a data verification task.
In the Information Entry step, configure the parameters and click Next step. You can select a service based on the data source settings that you configure. Then, select the required verification method and configure the related parameters.
In the Parsing results step, configure the parameters and click Next step.
In the Verification range confirmation step, confirm the information and click Determine.
3.3.3 Start the data verification task
After you create the data verification task, you can click All Initiate Validation to start all tasks. You can also select the data verification task in the task list and click Check Initiate Validation below the list to start the task.
After you start the data verification task, the task enters the Validation in state. You can refresh the page to view the execution result of the task.
If the result does not meet your business requirements, you can adjust the data and click RUN TASK to restart the task.
3.3.4 Change the agent
If you want to change the agent of the data verification plan, click Edit next to Agent List. In the popover that appears, select the required agent and click Determine. A data verification plan supports only one agent. If you select multiple agents, an error is reported.
The new agent checks its connectivity with the source and destination of the specified data verification plan. If no errors occur, the change is successful.