This topic describes the Full-Database Migration features, including its functions and limits.

Full-Database Migration is a convenient tool that can improve user efficiency and reduce user cost. It can quickly upload all tables in a MySQL database to MaxCompute simultaneously, which reduces time spent on creating batch tasks for the initial cloud migration.

For example, if a database contains 100 tables, the conventional method would require you to configure 100 data synchronization tasks. With Full-Database Migration, you can upload all tables at the same time. Because of the normalization design of the database tables, Full-Database Migration is subject to limitations and cannot guarantee all tables synchronized can be completed simultaneously to meet your business demands. 

Task generation rules

After you complete Full-Database Migration configuration, the MaxCompute tables are created and data synchronization tasks are generated based on the selected tables for synchronization.

The MaxCompute table names, field names, and field types are generated according to advanced settings. If no advanced settings are set, the MaxCompute table structures are the same to that of MySQL tables. The partition of these tables is pt, and in the format of yyyymmdd.

The generated data synchronization tasks are cyclic tasks scheduled on a daily basis. They run automatically in the morning the next day with a transfer rate of 1 MB/s. The actual synchronization task performances varies with the selected synchronization mode and concurrency settings. To customize generated tasks, click clone_database > Data Source Name > mysql2odps_table name in the synchronization task directory tree.
Note

We recommend that you perform a smoke test on the data synchronization tasks. You can find all synchronization tasks generated by a data source in project_etl_start > Full-Database Migration > Data Source Name under O&M Center > Task Management, and then right-click the test's corresponding task nodes.

Limits

Full-Database Migration is subject to certain limitations, due to the normalization design of the database tables. The limitations include:
  • Currently, only Full-Database Migration from the MySQL data source to MaxCompute is supported. The migration feature for Hadoop/Hive and Oracle data sources are under development.
  • Only daily incremental data load and full load modes are available.
    In case you need to synchronize historical data at a certain time period, and this feature cannot meet your requirements. The following are a few recommended solutions:
    • You can configure daily tasks instead of synchronizing historical data . You can call the provided supplementary data to trace historical data, which removes the need to run temporary SQL tasks to split data after the historical data is fully synchronized.
    • If you need to synchronize historical data, you can configure a task on the task development page, and click Run.  Then convert the data with SQL statements. These are both one-time operations.
    In case the daily incremental data load has a special business logic and cannot be identified by a date field, the following are some recommended solutions:.
    • The incremental data load can be achieved either through binlog (available in the DTS product) , or by modifying the data date field in the database.

      Currently, Data Integration supports the latter method, thus your database must contain the date field of the modified data . The system can detect if the data is modified on the same day as the business date with this field. If yes, all modified data is synchronized.

    • To facilitate incremental data load, we recommend that you include the gmt_create and gmt_modify fields when creating the database tables. Meanwhile, you can set the id field as the primary key to improve the table efficiency.
  • Full-Database Migration supports Upload in Batches and Upload All.
    Upload in Batches is configured with time intervals. Currently, the connection pool protection function for data sources is not supported, and is still under development.
    • To prevent the database from being overloaded, the Full-Database Migration provides an Upload in Batches mode, which allows you to split tables in batches at a time interval and prevents database overload from compromising service functionality. The following are two recommended solutions:
      • If you have primary and standby databases, we recommend that you synchronize the standby database data.
      • In batch tasks, each table has a database connection with the maximum speed of 1 Mbit/s. If you run synchronization tasks for 100 tables simultaneously, it will establish 100 database connections. We recommend that you select a number of concurrencies based on business conditions.
    • This feature does not support setting a specific task transfer rate, and the maximum speed of any generated task is 1 Mbit/s.
  • Only the mapping of all table names, field names, and field types are supported.
    During Full-Database Migration, the MaxCompute tables are created automatically. The table partition field is pt, the field type is string, and the format is yyyymmdd.
    Note When you select tables for synchronization, all fields cannot be edited and must be synchronized.