Use scheduling and overwrite policies to implement incremental migration.
Overview
In each task run, Data Online Migration performs a full scan of the source data address. If new data is generated during migration, it is migrated in the next run after the scan.
Data Online Migration cannot guarantee near-real-time incremental migration. This solution is not suitable if you migrate large numbers of files and require strict timeliness.
Data consistency is not guaranteed during migration. After the task is complete, verify migrated data at the destination data address. Before you delete source data, make sure destination data is correct. Otherwise, you are responsible for losses caused by data loss.
Migration tutorials
To migrate incremental data in different scenarios, combine scheduling and overwrite policies. See Migration tutorials and choose the tutorial that matches your scenario.
When you create a migration task, configure the Execution Time parameter for scheduling and the Overwrite Method parameter for overwrite behavior. The following table describes both policies.
Policy | Description |
Overwrite policy | Valid values of the Overwrite Method parameter are Do not overwrite, Overwrite All, and Overwrite based on the last modification time. For incremental migration, select Overwrite based on the last modification time. If a destination file meets the skip conditions, the source file with the same name is not overwritten. Note In this case, the system still requests file metadata from both the source and destination data addresses. |
Scheduling policy | A migration task can run in a specified time window or by execution frequency. See Supported execution frequencies. The execution ID increases by 1 each time the task runs, until the configured execution count is reached. The system schedules runs automatically by start time and pause time. After the configured run count is reached, automatic scheduling stops, but you can still start the task manually. |
In each run, the system performs a full scan. If a file already exists at the destination data address, the system uses the overwrite policy based on the last modification time to decide whether to skip the source file with the same name. This behavior enables incremental migration.
For details about scheduling by time window or frequency, see the Execution Time description in the "Create a migration task" section of "Migrate data" in migration tutorials.
For details about overwrite behavior, see the Overwrite Method description in the "Create a migration task" section of "Migrate data" in migration tutorials.
Example
Migrate incremental data at 00:00 every day
In this example, the source Object Storage Service (OSS) bucket stores 2,000 historical files, and about 50 new files are generated each day.
To migrate incremental files to the destination OSS bucket at 00:00 every day, perform the following steps:
Create a migration task. Set Overwrite Method to Overwrite based on the last modification time. Set Execution Time to Periodic Scheduling. Set Execution Frequency to Every Day. Set the start time to 00:00 and Executions to 100.
After configuration, Data Online Migration starts the task automatically at 00:00 every day, performs a full source scan, and migrates incremental data based on the overwrite policy using last modification time. Automatic scheduling stops after 100 runs, but you can still start the task manually.
If a large number of files must be migrated and one run cannot finish within a day, the task may still be running at 00:00 on the next day. In this case, Data Online Migration attempts to start it again at 00:00 on the following day.