All Products
Search
Document Center

ApsaraDB for OceanBase:View the details of a data migration project

Last Updated:Mar 19, 2024

After a data migration project starts, you can view the project metrics on the details page of the project, such as the basic information, and project progress and status.

Access the details page

On the details page of a data migration project, you can view the progress and status of the scheduled data migration project at different stages. This helps you learn the overall progress of data migration and the migration status of each table and object.

  1. Log on to the ApsaraDB for OceanBase console.

  2. In the left-side navigation pane, choose Data Transmission > Data Migration.

  3. On the Migration Projects page, click the name of the project. On the details page that appears, view the basic information and migration details of the project.

    image.png

    On the Data Migration page, you can filter data migration projects by status and keyword. A data migration project can be in one of the following states:

    • Not Configured: The data migration project has not been configured. You can click Configure in the Actions column to configure the project.

    • Not Started: The data migration project has not been started. You can click Start in the Actions column to start the project.

    • Running: The data migration project is in progress. You can view the data migration plan and current progress on the right and perform corresponding operations in the Actions column.

    • Modifying: The migration objects of the data migration project are being modified.

    • Integrating: The data migration project is being integrated with the corresponding migration object modification task.

    • Paused: The data migration project is manually paused. You can click Resume in the Actions column to resume the project.

      Note

      A data migration project remaining in the Paused state for a long time may not be able to be resumed due to the retention period of incremental logs.

    • Failed: The data migration project failed. You can locate the stage where the failure occurred on the right. To view the error message, click the project name to go to the project details page.

    • Completed: The data migration project is completed and the data transmission service has migrated the specified data to the destination database in the configured migration mode.

    • Stopping: The data migration project is being stopped. You cannot edit a data migration project in this state.

    • Stopped: The data migration project is stopped.

    You can change the name of a project in the Not Started, Running, Paused, or Failed state. For more information, see Change the name of a data migration project.

View basic information

The Basic Information section displays the basic information of the current data migration project.

image.png

Parameter

Description

ID

The unique ID of the data migration project.

Migration Type

The migration type specified when the data migration project was configured.

Alert Status

The alert status of the data migration project. We recommend that you enable alert monitoring to keep track of the project status. For more information, see Configure alerts.

Important

You cannot configure or modify alert monitoring for a data migration project in the Not Configured, Completed, Stopping, or Stopped state.

Created At

The time when the data migration project was created.

Connection Details

Click Connection Details to view the information about the source and destination databases of the data migration project. You can also copy the data source identifiers.

Instance Specification

The instance specifications specified when the data migration project was purchased. For more information, see Data transmission specifications.

You can perform the following operations:

  • View migration objects

    Click View Objects in the upper-right corner. The migration objects of the current data migration project are displayed. You can modify the migration objects when the project is running. For more information, see View and modify migration objects.

  • View or modify parameter configurations

    • For a data migration project in the Running state, click the More icon in the upper-right corner and then select View Parameter Configurations from the drop-down list to view the parameter configurations specified when the data migration project was created.

    • For a data migration project in the Not Started, Paused, or Failed status, click the More icon in the upper-right corner and select Modify Parameter Configurations from the drop-down list. In the Modify Parameter Configurations dialog box, modify the parameters and click OK.

    The parameters that can be modified vary with the type of the data migration project and the stage of the task.

  • Download Project Configuration

    The data transmission service allows you to download the configurations of data migration projects in batches. For more information, see Download and import settings of migration objects.

View migration details

The Migration Details section displays the status, progress, start time, completion time, and total time spent of all subtasks of the current project.

Schema migration

The definitions of data objects, such as tables, indexes, constraints, comments, and views, are migrated from the source database to the destination database. Temporary tables are automatically filtered out. If the source database is not an OceanBase database, the data transmission service performs format conversion and encapsulation based on the syntax definition and standard of the type of the destination tenant of OceanBase Database and then replicates the data to the destination database.

When you advance to the forward switchover stage in a data migration project, the hidden columns and unique indexes will be automatically dropped based on the type of the data migration project. For more information, see Schema migration mechanisms.

You can view the status, start time, completion time, total amount of time consumed, and table and view migration progress for a schema migration project on the Schema Migration page.

image.png

You can also perform the following operations on an object:

  • View Creation Syntax: On the Database or table tab, click View in the Actions column of an object to view the database or table creation syntax.

    Compatible DDL syntax executed on the OBServer node is displayed. Incompatible syntax is converted before it is displayed.

  • Modify Creation Syntax and Try Again: View the error information, check and modify the definition of the conversion result of a failed DDL statement, and then migrate the data to the destination again.

  • Retry/Retry All Failed Objects: You can retry failed schema migration tasks one by one or retry all failed tasks at a time.

  • Skip/Batch Skip: You can skip failed schema migration tasks one by one or skip multiple failed tasks at a time. To skip multiple objects at a time, click Batch Skip in the upper-right corner.

  • Remove/Batch Remove: You can remove failed schema migration tasks one by one or remove multiple failed tasks at a time. To remove multiple failed tasks at a time, click Batch Remove in the upper-right corner.

  • View Database Return Code: You can view the DDL statements executed on the OBServer node and the execution error information of a failed schema migration task.

Full data migration

The existing data is migrated from tables in the source database to the corresponding tables in the destination database. On the Full Data Migration page, you can filter objects by source and destination databases, or select View Objects with Errors to filter out objects that hinder the overall migration progress. You can also view related information on the Table Objects, Table Indexes, and Full Migration Performance tabs. The status of a full migration task changes to Completed only after the table objects and table indexes are migrated.

image.png

  • On the Table Objects tab, you can view the names, source and destination databases, estimated data volume, migrated data volume, and status of tables.

  • On the Table Indexes tab, you can view the objects, source and destination databases, creation time, end time, amount of time consumed, and status of tables. You can also view the index creation syntax, retry a single failed object or all failed indexes, and skip or remove one or more failed indexes.

  • On the Migration Performance tab, you can view the graphs of performance metrics such as the requests per second (RPS) and migration traffic of the source and destination, average read time and slicing time at the source, average write time at the destination, and performance benchmark, to identify performance issues in a timely manner.

    image.png

    You can combine full migration with incremental synchronization to ensure data consistency between the source and destination databases. If any objects fail to be migrated during full migration, the causes of the failure are displayed.

    Important

    If you did not select Schema Migration for Migration Type, the fields in the source database that match those in the destination database are migrated during full migration. The system does not check whether the table structures are consistent.

Incremental synchronization

After incremental synchronization starts, the data migration service synchronizes the data that has been changed (added, modified, or deleted) in the source database to the corresponding tables in the destination database. When services continuously write data to the source database, the data transmission service starts the incremental data pull module to pull incremental data from the source instance, parses and encapsulates the incremental data, and then stores the data. After that, the data transmission service starts the full data migration.

After the full data migration task is completed, the data transmission service starts the incremental data replay module to pull incremental data from the incremental data pull module. The incremental data is synchronized to the destination instance after being filtered, mapped, and converted. If you execute a DDL statement in the source database and cause a JDBCWriter exception, which then causes the data migration project to fail, the DDL statement and the Skip button are displayed. You can click Skip and confirm your operation.

Important

This operation may lead to data structure inconsistency between the source and destination databases. Proceed with caution.

When you create a data migration project, we recommend that you enable alert monitoring to monitor the project status in real time. For more information, see Configure alerts. If alert monitoring is disabled, when the incremental synchronization latency is less than or equal to 1 minute, the incremental synchronization status changes from Running to Monitoring by default.

image.png

Take note of the following items if alert monitoring is enabled:

  • When the incremental synchronization latency exceeds the specified alert threshold, the incremental synchronization status stays at Running and the system does not trigger any alerts.

  • When the incremental synchronization latency is less than or equal to the specified alert threshold, the incremental synchronization status changes from Running to Monitoring. After the incremental synchronization status changes to Monitoring, it will not change back to Running when the latency exceeds the specified alert threshold.

For a Running data migration project, you can view its latency, current timestamp, and incremental synchronization performance in the incremental synchronization section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a Paused or Failed data migration project, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled, You can also view the specific information about incremental synchronization objects and the incremental synchronization performance.

  • The Synchronization Object Statistics tab displays the statistics about table-level DML statements executed for each incremental synchronization object in the current project. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Incremental Synchronization Objects tab are the sum of the corresponding columns on this tab.

    统计数量

    If you enable the DDL/DML statistic feature and then disable it, the data transmission service will record the historical statistics.

  • The Incremental Synchronization Performance tab displays the following content:

    image

    • Latency: the latency in synchronizing incremental data from the source database to the destination database, in seconds.

    • Migration traffic: the traffic throughput of incremental data synchronization from the source database to the destination database, in Kbit/s.

    • Average execution time: the average execution time of an SQL statement, in ms.

    • Average commit time: the average commit time of a transaction, in ms.

    • RPS: the number of records processed per second.

Full verification

After the full data migration and incremental data migration are completed, the data transmission service automatically initiates a full data verification task to verify the data tables in the source and destination data sources.

The data transmission service also provides the corresponding APIs for you to initiate custom data verification tasks during incremental data synchronization.

On the Full Verification page, you can view the overall status, start time, end time, total amount of time consumed, estimated total number of rows, number of verified rows, RPS, and real-time traffic of the full verification task.

image.png

The Full Verification page contains the Verified Objects and Verification Performance tabs.

  • On the Verified Objects tab, you can view the verification progress and verification object list.

    image.png

    • You can view the names, source and destination databases, full data verification progress and results, and result summary of all migration objects.

    • You can filter migration objects by source database, destination database, or verification result.

    • You can select View Completed Objects Only to view the basic information of objects that have completed schema migration, such as the object names.

    • You can choose Re-verify > Restart Full Verification to run a full verification again for all migration objects.

    • Take note of the following items for tables with inconsistent verification results:

      If you need to reverify all data in the tables, choose Re-verify > Reverify Abnormal Table.

      If you only need to reverify inconsistent data, choose Re-verify > Verify Inconsistent Objects.

      Important

      Correction operations are not supported if the source database has no corresponding data.

  • On the Verification Performance tab, you can view the graphs of RPS and migration traffic of the source and destination databases, performance benchmark, and other performance statistics to identify performance issues in a timely manner.

The data transmission service allows you to skip full verification for a project that is being verified or has failed verification. On the Full Verification page, click Skip Full Verification. In the dialog box that appears, click OK.

Important

If you skip full data verification, you cannot resume the verification task for data comparison and correction. You can only clone the current project to initiate full data verification again. Therefore, proceed with caution.

After the full verification is completed, you can click Go To Next Stage to start a forward switchover. After you enter the switchover process, you cannot recheck the current verification task to compare or correct data.

image.png

Forward switchover

Forward switchover is an abstract and standard process of traditional system cutover and does not involve the switchover of application connections. This process includes a series of tasks that are performed for application switchover in a data migration project. You must make sure that the entire forward switchover process is completed before the application connections are switched over to the destination database.

Forward switchover is required for data migration. The data transmission service can ensure the completion of forward data migration in this process, and you can start the reverse incremental synchronization component based on your business needs. The forward switchover process involves the following operations:

  1. You must make sure that data migration is completed and wait until forward synchronization is completed.

  2. The data transmission service automatically supplements the check constraints and FOREIGN KEY constraints that are ignored in schema migration when you migrate data to an Oracle database or an Oracle tenant of OceanBase Database.

  3. The data transmission service automatically deletes the hidden columns and unique indexes on which the migration depends.

    This operation is required only when you migrate data between an Oracle database and an OceanBase database or between OceanBase databases. For more information, see Mechanisms for handling hidden columns.

  4. You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the destination database.

  5. You must disable triggers and FOREIGN KEY constraints in the source database. This operation is required only when the data migration project involves reverse incremental synchronization.

The forward switchover process contains the following steps:

image.png

  1. Start forward switchover.

    In this step, you can start forward switchover, but no operation is performed in the background. If you make sure that data migration is completed, you can click Start Forward Switchover to start the process.

    Important

    Before you start forward switchover, make sure that data writing has stopped in the source database.

  2. Perform a switchover precheck.

    In this step, the data transmission service checks the following items:

    • Synchronization latency between the source and destination databases: If the synchronization latency is within 15 seconds, this check item is passed.

    • Write privilege of the account in the source database. If the data migration project involves reverse incremental synchronization, the data transmission service additionally checks whether the account configured in the source database has the privilege to write data, to ensure that data can be properly written during reverse incremental synchronization.

    • Privilege of the account in the destination database for incremental data read. If the data migration project involves reverse incremental synchronization, the data transmission service additionally checks whether the account configured in the destination database has the privilege to read data, to ensure that data can be properly written to the destination database during reverse incremental synchronization.

    • Incremental logs in the destination database. If the data migration project involves reverse incremental synchronization, the data transmission service additionally checks whether the incremental logging configuration in the destination database meets the log extraction requirements of reverse incremental synchronization.

    If the switchover precheck is passed, the data transmission service automatically performs the next step. If the precheck fails, you can click Retry or Skip.

    Important

    If you click Skip, data loss may occur in the destination database, or the reverse incremental synchronization process may fail. Proceed with caution.

  3. Start the destination store.

    Note

    This step is available only when the data migration project involves reverse incremental synchronization.

    If the precheck for forward switchover is passed, the data transmission service automatically starts incremental log pulling for the destination database. This way, the data transmission service obtains DML and DDL operations in the destination database and parse and save related log data to prepare for reverse incremental synchronization. This step takes about 3 to 5 minutes.

  4. Confirm that data writing has stopped in the source database.

    In this step, the data transmission service checks whether business data is still being written to the source database. If you make sure that no new data is written to the source database, click OK to go to the next step.

  5. Confirm the data writing stop timestamp upon synchronization completion.

    In this step, the data transmission service checks whether the destination database is synchronized to the data writing stop timestamp in the source database. If not, the data transmission service continues to check until the destination database is synchronized to the timestamp. This way, the data transmission service makes sure that all data in the destination database is updated.

  6. Stop forward synchronization.

    In this step, you can stop forward synchronization. After forward synchronization is stopped, any database changes in the source database will no longer be synchronized to the destination database. If forward synchronization fails to be stopped, you can click Retry or Skip.

    Important

    You can click Skip only when you make sure that forward synchronization is completed in the background. Otherwise, data in the source database may be unexpectedly written to the destination database. Proceed with caution.

  7. Process database objects

    In this step, you can process the objects that are ignored in data migration or not supported by the data transmission service, to ensure normal operation of your business after the switchover to the destination database.

    • Migrate database objects to the destination database: You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the destination database. After you complete the migration, click Mark as Complete.

    • Disable triggers and FOREIGN KEY constraints in the source database: This operation is required only when the data migration project involves reverse incremental synchronization. It prevents data from being affected by triggers or FOREIGN KEY constraints, to avoid failures of reverse incremental synchronization. After you complete this operation, click Mark as Complete.

    • Supplement the objects ignored in schema migration to the destination database: This operation is required only when you migrate data to an Oracle database or an Oracle tenant of OceanBase Database. The data transmission service automatically supplements the objects that are ignored in schema migration to the destination database, such as check constraints and FOREIGN KEY constraints. When you migrate data to other types of databases, the preceding objects are migrated during schema migration by default.

    • Delete hidden columns and unique indexes added by the data transmission service: This operation is required only when you migrate data between an Oracle database and an OceanBase database or between OceanBase databases. The data transmission service automatically deletes the hidden columns and unique indexes that are added to the destination database to ensure data consistency. This operation runs automatically, and the amount of time required depends on the amount of data in the destination database. You can click Skip to skip this operation, but you must manually perform it later. Proceed with caution. For more information, see Mechanisms for handling hidden columns.

  8. Start reverse incremental synchronization.

    Note

    This step is available only when the data migration project involves reverse incremental synchronization.

    In this step, you can start incremental synchronization for the destination database to synchronize incremental DML or DDL operations from the destination database to the source database in real time. The configuration of incremental synchronization is the same as that specified when the project was created. For more information, see Incremental synchronization of DDL operations.

Reverse incremental synchronization

Note

Reverse incremental synchronization is not supported when you migrate data from an Oracle database to an Oracle tenant of OceanBase Database.

For a Running data migration project, you can view its latency, current timestamp, and performance of reverse incremental synchronization in the Reverse Incremental Migration section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a Paused or Failed data migration project, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled, You can also view the specific information about the objects and performance of reverse incremental synchronization.

image.png

  • The Synchronization Object Statistics tab displays the statistics about table-level DML statements executed for each incremental synchronization object in the current project. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Synchronization Object Statistics tab are the sums of the corresponding columns on this tab.

  • The Reverse Incremental Migration Performance tab displays the following content:

    • Latency: the latency in synchronizing incremental data from the destination database to the source database, in seconds.

    • Migration traffic: the traffic throughput of incremental data synchronization from the destination database to the source database, in Kbit/s.

    • Average execution time: the average execution time of an SQL statement, in ms.

    • Average commit time: the average commit time of a transaction, in ms.

    • RPS: the number of records processed per second.