Combine a scheduling policy with an overwrite policy to automatically migrate incremental data on a recurring schedule.
Overview
Each time a migration task runs, Data Online Migration performs a full scan of the source data. New or updated files are detected and migrated in the next scheduled run.
Data Online Migration does not guarantee real-time incremental migration. This solution is not suitable for scenarios with many files or strict timeliness requirements.
Data consistency is not guaranteed during migration. After a task completes, verify the destination data before deleting the source data. You are responsible for any data loss caused by premature deletion.
Migration implementation
Combine a scheduling policy and an overwrite policy for incremental migration. Select the tutorial for your scenario from Migration tutorials.
The following table describes these policies:
|
Policy |
Description |
|
Overwrite policy |
Three overwrite methods are available: Do not overwrite, Overwrite All, and Overwrite based on the last modification time. For incremental migration, select Overwrite based on the last modification time. The service skips source files whose destination counterparts already meet the skip conditions. Note
The Overwrite based on the last modification time policy still requests file metadata from both the source and destination. |
|
Scheduling policy |
Configure a task to run once or on a recurring schedule. The task runs automatically up to the specified execution count. After the limit is reached, automatic scheduling stops, but you can still start the task manually. Execution frequency reference. |
During each scheduled run, Data Online Migration performs a full scan of the source. If a same-name file exists at the destination, the overwrite policy determines whether to skip it based on last modification time.
Configure Execution Time and Overwrite Method in the "Create a migration task" section of your migration tutorial.
Migration example
Example: Daily incremental migration at 00:00
A source OSS bucket contains 2,000 historical files and receives approximately 50 new files daily. You want to migrate only the new files to a destination OSS bucket at 00:00 each day.
Follow these steps based on the OSS migration tutorial:
-
Create a migration task. For the overwrite mode, select Overwrite based on the last modification time. For task scheduling, select Periodic Scheduling, set the Execution frequency to run at midnight every day, and set the Number of executions to 100.
With this configuration, the task starts automatically at 00:00 daily for up to 100 runs. Each run performs a full scan and migrates only incremental data based on last modification time. After 100 runs, automatic scheduling stops, but you can still start the task manually.
If a single run takes more than 24 hours and is still running at the next scheduled start time (00:00), the service skips that run and retries at 00:00 the following day.