Validate data consistency with NimoFullCheck - ApsaraDB for MongoDB

NimoFullCheck is a tool developed by Alibaba Cloud to validate data consistency between a source Amazon DynamoDB and a destination ApsaraDB for MongoDB. This topic describes how to use NimoFullCheck to validate data consistency.

Prerequisites

You have migrated data from Amazon DynamoDB to ApsaraDB for MongoDB using NimoShake. For more information, see Use NimoShake to migrate Amazon DynamoDB to Alibaba Cloud.

Background

NimoFullCheck is used after a NimoShake migration to validate data consistency between the source Amazon DynamoDB and the destination ApsaraDB for MongoDB.

The validation process consists of the following two parts:

Brief check: Compares the item count in the Amazon DynamoDB table with the document count in the ApsaraDB for MongoDB collection. If the counts do not match, the process exits and reports an error.
Precise check: If the brief check passes, the precise check begins. The tool fetches and parses data from the source. If a unique index exists, the tool uses it to verify data consistency with ApsaraDB for MongoDB. If no unique index is found, the tool compares all data items at both the source and destination one by one, which can be slower.

Usage notes

NimoFullCheck supports consistency validation only for full data migration. If you run a validation during the incremental data synchronization phase, the tool reports the data as inconsistent.
NimoFullCheck performs a one-way validation. It uses the data in the destination ApsaraDB for MongoDB as the baseline to validate the data in the source Amazon DynamoDB.

Procedure

The following steps show how to use NimoFullCheck on Ubuntu to validate data consistency between Amazon DynamoDB and ApsaraDB for MongoDB.

Run the following command to download the NimoShake package.
```
wget https://github.com/alibaba/NimoShake/releases/download/release-v1.0.0-20191015/nimo.tar.gz
```
Note
We recommend that you download the latest version of the NimoShake package. For the download link, see NimoShake.
Run the following command to decompress the NimoShake package.
```
tar zxvf nimo.tar.gz
```
After decompressing the package, run the cd nimo command to enter the nimo directory.

Run the following command with the required parameters to start the validation.

./nimo-full-check.linux --<parameter>=<value> --<parameter>=<value>

The following table describes the parameters.

Parameter	Description	Example
id	The migration task ID. This ID must match the one specified for the migration task in NimoShake. For more information, see Use NimoShake to migrate Amazon DynamoDB to Alibaba Cloud.	`--id=nimo-shake`
logLevel	The log level. Valid values: none: No logs are printed. error: Logs that contain error-level information. warn: Logs that contain warning-level information. info: Logs that reflect the current system status. debug: Logs that contain debugging information. Default value: info.	`--logLevel=info`
sourceAccessKeyID	The access key ID for the source Amazon DynamoDB.	`--sourceAccessKeyID=xxxxxxxxxx`
sourceSecretAccessKey	The secret access key for the source Amazon DynamoDB.	`--sourceSecretAccessKey=xxxxxxxxxx`
sourceSessionToken	The session token for the source Amazon DynamoDB. This parameter is optional.	`--sourceSessionToken=xxxxxxxxxx`
sourceRegion	The region of the source Amazon DynamoDB instance. This parameter is optional.	`--sourceRegion=us-east-2`
qpsFull	The frequency of the `Scan` command on a table, which specifies the maximum number of times `Scan` can be called per second. Default value: 10000.	`--qpsFull=10000`
qpsFullBatchNum	The number of data entries to fetch in each Scan operation. Default: 128.	`--qpsFullBatchNum=128`
targetAddress	The endpoint of the destination ApsaraDB for MongoDB instance. To view the endpoint, see Connect to a replica set instance or Connect to a sharded cluster instance. Example: mongodb://username:password@s-*****-pub.mongodb.rds.aliyuncs.com:3717	`--targetAddress=mongodb://username:password@s-*****-pub.mongodb.rds.aliyuncs.com:3717`
diffOutputFile	The path and name of the output file for validation differences. If not specified, it defaults to `nimo-full-check-diff`.	`--diffOutputFile=nimo-full-check-diff`
parallel	The number of threads to use for the consistency validation. Default value: 16.	`--parallel=16`
sample	The number of documents to sample from each table for validation. A value of `0` validates all documents. The default value is `1000`, which means a maximum of 1,000 documents are sampled from a table for comparison.	`--sample=1000`
filterCollectionWhite	A whitelist of collections to validate. Only the specified collections are validated. For example, `--filterCollectionWhite=c1;c2` validates only collections c1 and c2.	`--filterCollectionWhite=c1;c2`
filterCollectionBlack	A blacklist of collections to exclude from validation. For example, `--filterCollectionBlack=c1;c2` indicates that all collections except c1 and c2 are validated.	`--filterCollectionBlack=c1;c2`
convertType	Specifies whether DynamoDB data types were converted during the migration phase. Valid values: raw: Data is written directly without conversion. change: Data is converted. For example, `{"hello":"1"}` is converted to `{"hello": 1}`. Note This value must match the convertType setting used during the migration. A mismatch will cause the validation to fail.	`--convertType=change`
version	Prints the current version number of NimoFullCheck. Note This parameter is a flag that takes no value. When you include the `--version` flag, the tool prints its version number and exits.	`--version`
help	Displays all supported parameters for NimoFullCheck.	`--help`

Note

Upon successful validation, the message full check done! is displayed. If a fatal error occurs, the program exits and prints an error message.