ApsaraDB for MongoDB:Restore a cloud disk backup to a self-managed database

Usage notes

• Use a mongorestore version that is compatible with your MongoDB database. Due to frequent updates, older mongorestore versions may be incompatible with newer database versions. For more information, see mongorestore.

• Even if a collection contains very little data and has only one BSON file, such as myDatabase/myCollection/data/myCollection_0_part0.bson, you must merge or rename the BSON files. This is because mongorestore processes BSON files based on their filename prefixes.

• When you download a cloud disk backup, the download process also handles empty collections with preserved schemas, resulting in an empty BSON file that contains database and collection name information. The mongorestore tool can handle these empty files correctly.

• For a sharded cluster instance, the downloaded cloud disk backup file no longer contains sharding route information. Therefore, you can restore the backup data to any single-node, replica set, or sharded cluster instance. If you want to restore the data to a sharded cluster instance, you must perform pre-sharding.

Prerequisites

• On the client (a local server or an ECS instance) that hosts your self-managed MongoDB database, install a MongoDB version that matches the version of your ApsaraDB for MongoDB instance. For installation instructions, see Install MongoDB.

• You have downloaded the logical backup file. If not, see Download backup files.

Procedure

1. Copy the downloaded backup file to the client that hosts both the mongorestore tool and your self-managed MongoDB database.

2. Decompress the backup file.

   You can download backup files in tar.zst or tar.gz format, which use the zstd and gzip compression algorithms, respectively. You can select the download format by using the UseZstd parameter of the CreateDownload OpenAPI operation.

   tar.zst (Console)

   zstd -d -c <backup_file.tar.zst> | tar -xvf - -C <extraction_directory>

   Ensure that the zstd tool is installed on the client and that the extraction directory exists.

   Example:

   mkdir -p ./download_test/test1
   zstd -d -c test1.tar.zst | tar -xvf - -C /Users/xxx/Desktop/download_test/test1/

   tar.gz (API)

   tar -zxvf <backup_file.tar.gz> -C <extraction_directory>

   Ensure that the extraction directory exists.

   Example:

   mkdir -p ./download_test/test1
   tar -zxvf testDB.tar.gz -C /Users/xxx/Desktop/download_test/test1/

3. Merge the BSON files.

   On a client with a Python environment, copy the following script and save it as merge_bson_files.py.

   import os
   import struct
   import sys
   import argparse
   import shutil
   import re
   
   # Handle strings for compatibility with both Python 2 and 3
   if sys.version_info[0] >= 3:
       unicode = str
   
   
   def merge_single_bson_dir(input_dir: str, output_dir: str, namespace: str) -> None:
       """
       Merges BSON files within a single directory.
   
       Args:
           input_dir (str): The directory path containing the BSON files.
           output_dir (str): The directory path for the output file.
           namespace (str): The name of the output file (without extension).
       """
       try:
           # Get all BSON files matching the ***_*_part*.bson pattern and sort them by name
           files = [f for f in os.listdir(input_dir) if re.match(r'^.+_.+_part\d+\.bson$', f)]
           files.sort()  # Sort by filename
   
           if not files:
               print("No matching .bson files found in {}".format(input_dir))
               return
   
           output_file = os.path.join(output_dir, "{}.bson".format(namespace))
           if os.path.exists(output_file):
               print("Output file {} already exists, skipping...".format(output_file))
               return
   
           print("Merging {} files into {}...".format(len(files), output_file))
   
           # Stream-read and merge the files
           total_files = len(files)
           with open(output_file, "wb") as out_f:
               for index, filename in enumerate(files, 1):
                   file_path = os.path.join(input_dir, filename)
                   print("  Processing file {}/{}: {}...".format(index, total_files, filename))
   
                   try:
                       with open(file_path, "rb") as in_f:
                           while True:
                               # Read the BSON document size
                               size_data = in_f.read(4)
                               if not size_data or len(size_data) < 4:
                                   break
   
                               # Parse the document size (little-endian)
                               doc_size = struct.unpack("<i", size_data)[0]
   
                               # Reread the complete document data
                               in_f.seek(in_f.tell() - 4)
                               doc_data = in_f.read(doc_size)
   
                               if len(doc_data) != doc_size:
                                   break
   
                               out_f.write(doc_data)
                   except Exception as e:
                       print("Error reading {}: {}".format(filename, str(e)))
       except Exception as e:
           print("Error in merge_single_bson_dir: {}".format(str(e)))
   
   
   def merge_bson_files_recursive(input_root: str, output_root: str = None) -> None:
       """
       Recursively traverses directories and merges all BSON files.
   
       Args:
           input_root (str): The root directory path containing the BSON files.
           output_root (str): The root directory for the output files, defaults to input_root.
       """
       if output_root is None:
           output_root = input_root
   
       # Ensure the output root directory exists
       if not os.path.exists(output_root):
           os.makedirs(output_root)
   
       print("Scanning directories in {}...".format(input_root))
       
       # Traverse all items in the input root directory
       for item in os.listdir(input_root):
           item_path = os.path.join(input_root, item)
           
           # If it is a directory, process it
           if os.path.isdir(item_path):
               print("Processing directory: {}".format(item))
               
               # Create the corresponding output directory
               output_item_path = os.path.join(output_root, item)
               if not os.path.exists(output_item_path):
                   os.makedirs(output_item_path)
               
               # Traverse all subdirectories and files within this directory
               for item_d in os.listdir(item_path):
                   sub_item_path = os.path.join(item_path, item_d)
                   for sub_item in os.listdir(sub_item_path):
                       data_path = os.path.join(sub_item_path, sub_item)
                       # If it's a "data" directory, merge the BSON files within it
                       if os.path.isdir(data_path) and sub_item == "data":
                           # Extract the namespace (parent directory name)
                           namespace = os.path.basename(sub_item_path)
                           merge_single_bson_dir(data_path, output_item_path, namespace)
                       # If it's a .metadata.json file, copy it directly to the corresponding output directory
                       elif sub_item.endswith(".metadata.json"):
                           src_file = os.path.join(sub_item_path, sub_item)
                           target_dir = os.path.join(output_item_path, sub_item)
                           shutil.copy(src_file, target_dir)
                           print("Copied metadata file: {}".format(sub_item))
               print("Finished processing directory: {}".format(item))
   
   
   if __name__ == "__main__":
       parser = argparse.ArgumentParser(description="Recursively merge BSON files")
       parser.add_argument("input_root", help="The root directory path containing the BSON files")
       parser.add_argument("-o", "--output_root", help="The root directory path for output files, defaults to the input root")
   
       args = parser.parse_args()
       merge_bson_files_recursive(args.input_root, args.output_root)

   Run the command:

   python merge_bson_files.py <input_directory> -o <output_directory>

4. Use the mongorestore tool to restore the backup data to your self-managed database instance.

   # Restore a single collection
   mongorestore --uri=<mongodb-uri> --db <db> --collection <collection>  <xxx.bson>
   # Example for a single collection
   mongorestore --uri='mongodb://127.x.x.x:27017/?authSource=admin' --db testDB --collection coll1 ./testDB/coll1.bson 
   # Restore a single database
   mongorestore --uri=<mongodb-uri> --db <db> --dir </path/to/bson/dir>
   # Example for a single database
   mongorestore --uri='mongodb://127.x.x.x:27017/?authSource=admin' --db testDB --dir ./testDB 
   # Restore an entire instance
   mongorestore --uri=<mongodb-uri>  --dir </path/to/bson/dir>
   # Example for an entire instance
   mongorestore --uri='mongodb://127.x.x.x:27017/?authSource=admin' --dir ./

   Parameters:

   • <mongodb-uri>: The connection string URI of your self-managed database or ApsaraDB for MongoDB instance. The URI includes the username, password, server IP address, and port. For more information, see the official documentation.

   • <db>: The name of the database to restore.

   • <collection>: The name of the collection to restore.

   • <xxx.bson>: The BSON backup file for the single collection you want to restore.

   • <path/to/bson/dir>: The directory that contains the BSON files for the restore operation.

FAQ