NimoFullCheck is a tool developed by Alibaba Cloud to check data consistency between an Amazon DynamoDB database and an ApsaraDB for MongoDB database. This topic describes how to use NimoFullCheck to check data consistency after you migrate data from an Amazon DynamoDB database to an ApsaraDB for MongoDB database.
Prerequisites
Background information
After you migrate data from an Amazon DynamoDB database to an ApsaraDB for MongoDB database, you can use NimoFullCheck to check data consistency between the two databases.
The check consists of the following two steps:
- Brief check: checks whether the number of items in a table in the Amazon Dynamo database is equal to the number of documents in the corresponding collection in the ApsaraDB for MongoDB database. If the numbers are different, the check terminates and an error message is returned. You can locate issues based on the returned error message.
- Precise check: precisely compares data in the two databases after the brief check is passed. NimoFullCheck fetches data from the Amazon Dynamo database and parses the data. If the data contains unique indexes, NimoFullCheck compares the data with that in the destination ApsaraDB for MongoDB database based on the unique indexes. If the data does not contain unique indexes, NimoFullCheck compares all data entries in two databases one by one, which is slow.
Usage notes
- NimoFullCheck only supports consistency check for full data migration. If you check data consistency after an incremental data synchronization, the result is inconsistent.
- NimoFullCheck uses data in the ApsaraDB for MongoDB database as the baseline for check. In other words, NimoFullCheck checks whether data in the Amazon DynamoDB database is consistent with that in the ApsaraDB for MongoDB database.
Procedure
The following procedure assumes that you run NimoFullCheck in the Ubuntu operating system.
- Run the following command to start NimoFullCheck with required parameters:
The following table describes the parameters of NimoFullCheck../nimo-full-check.linux --<Parameter 1>=<Value 1> --<Parameter 2>=<Value 2>
Parameter Description Example id The ID of the migration task. Set the value to the ID of the migration task that is specified when you used NimoShake to migrate data. For more information, see Migrate an Amazon DynamoDB database to ApsaraDB for MongoDB by using NimoShake. --id = nimo-shakelogLevel The level of the logs to be generated. Valid values: - none: does not generate logs.
- error: generates logs that contain error messages.
- warn: generates logs that contain warnings.
- info: generates logs that indicate system status.
- debug: generates logs that contain debugging information.
Default value: info.
--logLevel = infosourceAccessKeyID The access key ID used to connect to the source Amazon DynamoDB database. --sourceAccessKeyID = xxxxxxxxxxsourceSecretAccessKey The secret access key used to connect to the source Amazon DynamoDB database. --sourceSecretAccessKey = xxxxxxxxxxsourceSessionToken Optional. The session token used to access the source Amazon DynamoDB database. --sourceSessionToken = xxxxxxxxxxsourceRegion Optional. The region where the source Amazon DynamoDB database resides. --sourceRegion = us-east-2qpsFull The number of times that the Scancommand is run on tables per second.Default value: 10000.--qpsFull = 10000qpsFullBatchNum The number of data entries to fetch per second. Default value: 128. --qpsFullBatchNum = 128targetAddress The endpoint of the destination ApsaraDB for MongoDB database. For more information about how to view the endpoint, see Overview of replica set instance connections or Overview of sharded cluster instance connections. Example: mongodb://username:password@s-*****-pub.mongodb.rds.aliyuncs.com:3717.
--targetAddress = mongodb://username:password@s-*****-pub.mongodb.rds.aliyuncs.com:3717diffOutputFile The name of the file that stores the information about inconsistent data. If you do not specify this parameter, the default file name nimo-full-check-diffis used.--diffOutputFile = nimo-full-check-diffparallel The number of threads to be used for consistency check. Default value: 16. --parallel = 16sample The maximum number of documents to be checked in each collection. A value of 0 indicates that all documents in a collection are checked. Default value: 1000. A value of 1000 indicates that a maximum of 1,000 documents can be checked in a collection. --sample = 1000filterCollectionWhite The collection whitelist for consistency check. Set the value to the names of the collections that must be checked. Example: --filterCollectionWhite = c1;c2, indicating that only collections c1 and c2 are checked.--filterCollectionWhite = ci;c2filterCollectionBlack The collection blacklist for consistency check. Set the value to the names of the collections that do not need to be checked. Example: --filterCollectionBlack = c1;c2, indicating that all collections other than c1 and c2 are checked.--filterCollectionBlack = ci;c2convertType Specifies whether the source data, which uses the Dynamo protocol, was converted during the migration. Valid values: - raw: Data was written to the destination database without conversion.
- change: Data was converted before it is written to the destination database. For example,
{"hello":"1"}was converted to{"hello": 1}.
Note The value of the parameter must be the same as that specified for migration. If the values are different, the check fails.--convertType = changeversion Displays the version number of NimoFullCheck. Note You do not need to specify a value for the parameter. If you need to display the version number, add the--versionparameter to the command.--versionhelp Displays all the parameters that are supported by NimoFullCheck. --helpNote If the check is successful, the messagefull check done!is returned. If the check terminates due to an error, NimoFullCheck exits and an error message is returned. You can locate issues based the returned error message.