ApsaraDB for MongoDB:Restore a local disk backup to a self-managed database

Usage notes

• Because MongoDB is continuously updated, older versions of mongorestore are not compatible with newer versions of MongoDB. For information about how to select a compatible mongorestore version, see mongorestore.

• If your self-managed database is a sharded cluster, you must set the <hostname> parameter to the address of the mongos component in your self-managed database.

• If your self-managed database is a sharded cluster, you must add the --nsExclude="config.*" parameter to the restore command. Otherwise, errors may occur during the restore process.

• When you restore data from a sharded cluster instance to a self-managed database, you need to download the backup data for each shard of the cluster instance and import it into the self-managed database. If the sharded cluster instance contains orphaned documents, this may result in dirty data in the self-managed database.

Prerequisites

• Download and install a version of MongoDB that is the same as the database version of your ApsaraDB for MongoDB instance on the client where your self-managed MongoDB database is located. The client can be a local server or an ECS instance. For installation instructions, see Install MongoDB.

• You have downloaded the logical backup. If you have not, see Download backup files.

Procedure

1. Copy the downloaded backup file to the client that hosts your self-managed MongoDB database. This client must have the mongorestore tool installed.

2. Run the following command to import the data from the backup file to your self-managed MongoDB database.

   mongorestore -h <hostname> --port <server port> -u <username> -p <password> --drop --gzip --archive=<backupfile> -vvvv --stopOnError

   Modify the following parameters:

   • <hostname>: The server address of your self-managed MongoDB database. Use 127.0.0.1 for a local instance.

     If your self-managed database is a sharded cluster, set this parameter to the address of the mongos component.

   • <server port>: The port of your self-managed MongoDB database.

   • <username>: The username for logging in to your self-managed MongoDB database. Ensure the user has permissions for all databases. We recommend using the root account.

   • <password>: The password for the database user.

   • <backupfile>: The name of the downloaded logical backup file.

   The following command parameters do not need to be modified:

   • --drop: Drops each collection from the target database before restoring it from the backup.

     Note

     When restoring backups from multiple shards to the same sharded cluster, use this parameter only when restoring the backup of the first shard.

   • --gzip: Decompresses the backup file, since it is compressed with gzip.

     Note

     This parameter is supported starting from MongoDB 3.1.4. For more information, see mongo-tools.

   • -vvvv: Specifies the verbosity level of the output. More 'v' characters result in more detailed output.

   • --stopOnError: Stops the import process if an error occurs.

   • --nsExclude: Excludes matching collections from the restore. For example, --nsExclude="config.*".

   Example:

   mongorestore -h 127.0.0.1 --port 27017 -u root -p ******** --drop --gzip --archive=hins1111_data_20190710.ar -vvvv --stopOnError

Prerequisites

• The instance must be a replica set.

• The TDE feature must be disabled for the instance.

• The storage engine of the instance must be WiredTiger or RocksDB. If the storage engine is TerarkDB, see Restore data from a cloud disk backup.

  Note

  • You can view the storage engine of your instance on the Basic Information page in the ApsaraDB for MongoDB console.

  • If the storage engine of the instance is RocksDB, you must compile and install MongoDB with the RocksDB storage engine.

Database version requirements

The version of the ApsaraDB for MongoDB instance must be compatible with the version of the self-managed MongoDB database. The following table shows the version mappings.

Physical backup file formats

Environment

• The server used in this example is an Alibaba Cloud ECS instance running an Ubuntu 16.04 (64-bit) image. For more information about how to create an ECS instance, see Create an ECS instance by using the wizard.

• The corresponding version of MongoDB is installed on the ECS instance. For installation instructions, see the official MongoDB documentation.

• MongoDB environment variables are configured on the ECS instance. You do not need to enter the full path to executable files when running commands.

• The /test/mongo/data directory serves as the database directory for the physical restore.

• The /test/mongo/data1 and /test/mongo/data2 directories serve as the database directories for the replica set nodes.

Step 1: Configure environment variables

Configure environment variables for MongoDB in your self-managed environment to avoid typing full paths when running commands. Before you start, make sure you have installed MongoDB.

If you have already configured environment variables for MongoDB, skip this step and proceed to Step 2: Download and decompress the physical backup file.

1. Run the following command to open the profile file in Linux:

   sudo vi /etc/profile

2. Press i to enter insert mode, and add the following line to the end of the file:

   export PATH=$PATH:/<path_to_mongodb_server>/bin

   Note

   In this example, the MongoDB server path is /test/mongo/bin. Replace this with your actual path.

   Example:

   export PATH=$PATH:/test/mongo/bin

3. Press Esc, type :wq, and press Enter to save and exit.

4. Run the following command to apply the changes:

   source /etc/profile

Step 2: Download and decompress the backup file

1. Download the MongoDB physical backup file.

   wget -c '<public_download_URL_of_backup_file>' -O <custom_filename>.<extension>

   Example:

   wget -c 'http://rdsbak-hz-v3.oss-cn-hangzhou-internal.aliyuncs.com/custins5475****/hins1907****_data_20210906103710_qp.xb?Expires=......' -O backupfile._qp.xb

   Note

   • Based on the file type, ensure the file extension is either .tar.gz or _qp.xb.

   • Enclose the download URL in single quotation marks (') to ensure it is parsed correctly.

2. Run the following command to create a data directory in /test/mongo/ and move the downloaded MongoDB physical backup file to the /test/mongo/data/ directory.

   mkdir -p /test/mongo/data && mv <physical_backup_filename.extension> /test/mongo/data

3. Decompress the physical backup file.

   • If the downloaded physical backup file has a .tar.gz extension, such as hins20190412.tar.gz, use the following method to decompress it.

     cd /test/mongo/data/ && tar xzvf hins20190412.tar.gz 

   • If the downloaded physical backup file has a _qp.xb extension, such as hins20190412_qp.xb, use the following method to decompress it.

     1. Install the Percona XtraBackup tool and the qpress package. For more information, see the installation steps on the official Percona XtraBackup website.

     2. Decompress the physical backup file.

        # Navigate to the directory where the file is located.
        cd /test/mongo/data/
        # Unpack the file.
        cat hins20190412_qp.xb | xbstream -x -v
        # Decompress the physical backup file.
        innobackupex --decompress --remove-original /test/mongo/data

Step 3: Restore in standalone mode

1. Run the following command to create a configuration file named mongod.conf in the /test/mongo directory.

   touch /test/mongo/mongod.conf

2. Run vi /test/mongo/mongod.conf to open the mongod.conf file, then press i to enter insert mode.

   Select a startup configuration template based on the storage engine of your ApsaraDB for MongoDB instance and copy it into the mongod.conf file.

   Note

   The configuration file sets the startup mode to standalone and enables authentication.

   • WiredTiger storage engine

     systemLog:
         destination: file
         path: /test/mongo/mongod.log
         logAppend: true
     security:
         authorization: enabled
     storage:
         dbPath: /test/mongo/data
         directoryPerDB: true
     net:
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid

     Note

     ApsaraDB for MongoDB uses the WiredTiger storage engine by default and enables the directoryPerDB option. Therefore, this option is included in the configuration.

   • RocksDB storage engine

     systemLog:
         destination: file
         path: /test/mongo/logs/mongod.log
         logAppend: true
     security:
         authorization: enabled​
     storage:
         dbPath: /test/mongo/data
             engine: rocksdb
     net:
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid

3. Press Esc, type :wq, and press Enter to save and exit.

4. Start MongoDB with the new mongod.conf configuration file.

   mongod -f /test/mongo/mongod.conf

5. After the startup is complete, run the following command to log in to the MongoDB database.

   mongo --host 127.0.0.1 -u <username> -p <password> --authenticationDatabase admin

   • <username>: The database account for the MongoDB instance. The default is root.

   • <password>: The password for the database user.

     Note

     If your password contains special characters, enclose it in single quotation marks ('), for example: 'test123!@#'. Otherwise, the login may fail.

6. In the mongo shell, run show dbs to view all databases on your local MongoDB instance and verify that the restore was successful.

7. The restore process is now complete. You can run the exit command to quit the mongo shell.

After completing these steps, you can start the MongoDB database in standalone mode. To start the database in replica set mode, see Step 4.

Step 4: Start in replica set mode

By default, a physical backup from ApsaraDB for MongoDB contains the replica set configuration of the original instance. To start the database in replica set mode, you must first remove this configuration. Follow these steps:

1. From the command line, use the mongo shell to log in to the MongoDB database as the test user.

   mongo --host 127.0.0.1 -u test -p <test_user_password> --authenticationDatabase admin

   Note

   If your password contains special characters, enclose it in single quotation marks ('), for example: 'test123!@#'. Otherwise, the login may fail.

2. After logging in, run the commands in the code block below to perform the following actions:

   1. In the admin database, create a temporary user and grant it temporary read and write permissions on the local database.

   2. Switch to the temporary user and remove the original replica set configuration from the local database.

   3. Switch back to the test user and delete the temporary user and its permissions.

      Note

      Replace the <test_user_password> placeholder in the following code with your actual test user password before running it.

   use admin
   db.runCommand({
       createRole: "tmprole",
       roles: [
           {
               role: "test",
               db: "admin"
           }
       ],
       privileges: [
           {
               resource: {
                   db: 'local',
                   collection: 'system.replset'
               },
               actions: [
                   'remove'
               ]
           }
       ]
   })
   db.runCommand({
       createUser: "tmpuser",
       pwd: "tmppwd",
       roles: [
           'tmprole'
       ]
   })
   db.auth('tmpuser','tmppwd')
   use local
   db.system.replset.remove({})
   use admin
   db.auth('test','<test_user_password>')
   db.dropRole('tmprole')
   db.dropUser('tmpuser')

3. Run the following commands to shut down the MongoDB service and exit the mongo shell.

   use admin
   db.shutdownServer()
   exit

4. Create a replica set authentication file.

   To start MongoDB in replica set mode, you must create a keyfile that replica set members use for mutual authentication.

   1. Run the following command to create a keyFile directory in the mongo directory and create a key file within it.

      mkdir -p /test/mongo/keyFile && touch /test/mongo/keyFile/mongodb.key

   2. Run vi /test/mongo/keyFile/mongodb.key to open the key file. Press i to enter insert mode and input the key content. For example:

      MongoDB Encrypting File

      Note

      The keyfile content has the following restrictions:

      • Must be 6 to 1024 characters in length.

      • Can only contain characters from the base64 character set.

      • Cannot contain the equals sign (=).

   3. Press Esc, type :wq, and press Enter to save and exit.

   4. Run the following command to change the permissions of the authentication file to 400. This ensures that the file is readable only by its owner.

      sudo chmod 400 /test/mongo/keyFile/mongodb.key

   Note

   This authentication file will be used by all nodes in the replica set.

5. Follow these steps to prepare two empty nodes for the replica set.

   1. Run the following command to copy the mongod.conf file twice to serve as startup configuration files for the other two nodes.

      cp /test/mongo/mongod.conf /test/mongo/mongod1.conf && cp /test/mongo/mongod.conf /test/mongo/mongod2.conf

   2. Run the following command to create data directories for the other two nodes.

      mkdir -p /test/mongo/data1 && mkdir -p /test/mongo/data2

6. Modify the configuration file for each node as follows:

   • Run vi /test/mongo/mongod.conf to open the configuration file for Node 1. Update the file with the following content, then save and exit.

     systemLog:
         destination: file
         path: /test/mongo/mongod.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid
     replication:
         replSetName: "rs0"

   • Run vi /test/mongo/mongod1.conf to open the configuration file for Node 2. Update the file with the following content, then save and exit.

     systemLog:
         destination: file
         path: /test/mongo/mongod1.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data1
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27018
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod1.pid
     replication:
         replSetName: "rs0"

   • Run vi /test/mongo/mongod2.conf to open the configuration file for Node 3. Update the file with the following content, then save and exit.

     systemLog:
         destination: file
         path: /test/mongo/mongod2.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data2
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27019
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod2.pid
     replication:
         replSetName: "rs0"

   Key parameter descriptions:

   • systemLog.path / path: The path to the MongoDB log file for the current node,

   • storage.dbPath: The path to the MongoDB data files for the current node.

   • processManagement.pidFilePath: The path to the PID file (which stores the process ID) for the current node's MongoDB process.

   • security.keyFile: The path to the replica set authentication file. All nodes must use the same file.

   • net.bindIp: The IP address of the current node. If all nodes are deployed on the same server, they can use the same IP address.

   • net.port: The port number for the current node. If all nodes are deployed on the same server, they must use different port numbers.

   • replication: The replica set configuration.

   • replication.replSetName: The name of the replica set.

7. Run the following command to start the three nodes.

   mongod -f /test/mongo/mongod.conf && mongod -f /test/mongo/mongod1.conf && mongod -f /test/mongo/mongod2.conf

8. After the nodes have started, log in to the MongoDB database as the test user.

   mongo --host 127.0.0.1 -u test -p <test_account_password> --authenticationDatabase admin

   Note

   If your password contains special characters, enclose it in single quotation marks ('), for example: 'test123!@#'. Otherwise, the login may fail.

9. In the mongo shell, run the following command to add the nodes to the replica set and initialize it.

   rs.initiate( {
      _id : "rs0",
      version : 1,
      members: [
         { _id: 0, host: "127.0.0.1:27017" , priority : 1},
         { _id: 1, host: "127.0.0.1:27018" , priority : 0},
         { _id: 2, host: "127.0.0.1:27019" , priority : 0}
      ]
   })

   Successful initialization example:

   > rs.initiate( {
   ...    _id : "rs0",
   ...    version : 1,
   ...    members: [
   ...       { _id: 0, host: "127.0.0.1:27017" , priority : 1},
   ...       { _id: 1, host: "127.0.0.1:27018" , priority : 0},
   ...       { _id: 2, host: "127.0.0.1:27019" , priority : 0}
   ...    ]
   ... })
   { "ok" : 1 }
   rs0:SECONDARY>

   Note

   This step uses the rs.initiate() command. For detailed usage, see the official MongoDB documentation on rs.initiate().

   After the command runs successfully, the two new nodes will start synchronizing data from the primary node. The time required for this process varies depending on the size of the backup file. Once data synchronization is complete, the replica set is fully operational.

10. Follow these steps to verify that the startup was successful.

    1. Run exit to quit the mongo shell.

    2. Run the following command to log back in to the MongoDB database.

       mongo -u <username> -p <password> --authenticationDatabase admin

       • <username>: The database account for the MongoDB instance. The default is root.

       • <password>: The password for the database account.

         Note

         If your password contains special characters, enclose it in single quotation marks ('), for example: 'test123!@#'. Otherwise, the login may fail.

    3. Observe the left side of the mongo shell prompt. If it displays <replica_set_name>:PRIMARY>, the replica set has started successfully.