ApsaraDB for OceanBase:Configure matching rules

Background information

When you create a data migration or synchronization project, you must specify the objects to migrate or synchronize. The data transmission service allows you to directly specify objects, import objects, or configure matching rules for objects. You can configure wildcard-based matching rules to specify migration or synchronization objects. You can also configure object mapping logic between the source and the destination. This allows you to specify a large number of objects to be migrated or synchronized in a simple and efficient manner. New tables meeting the matching rules can be automatically synchronized to the destination through DDL operations for incremental synchronization. For more information, see Supported DDL operations for synchronization and limitations.  

The following table lists the wildcard-based matching rules supported for the data transmission service, and the requirements.

The requirements for matching rules are as follows:

• You cannot use wildcards for both database and table names at the destination, for example, kd_test*.person*=kd_test*.person*.

• If you use wildcards for both the source and destination databases, the database expression must be the same for the source and destination, indicating smooth database migration.

• If you use wildcards for both the source and destination tables, the table expression must be the same for the source and destination, indicating smooth table migration.

• If you use a wildcard for databases at the destination, you must also use a wildcard for databases at the source.

• If you use a wildcard for tables at the destination, you must also use a wildcard for tables at the source.

Limitations

• In a data synchronization project, you can configure matching rules to select synchronization objects only for synchronization between databases.

• The data transmission service supports multiple rules. Make sure that each rule is placed in a single row and has no leading or trailing spaces.

• Matching rules for migration or synchronization objects must not be empty. Object exclusion rules can be empty.

• DDL operations for change are not supported during schema migration and full migration.

• When you configure matching rules to select migration or synchronization objects, table names cannot contain line breaks, spaces, and the following special characters: . | " ' ` ( ) = ; / & * ? [] [!].

• You are not allowed to configure multiple matching rules to map different tables in the same database at the source to different databases at the destination, for example, a.a* = b.a* & a.b* = c.b*.

• In a scenario of database or table aggregation, reverse incremental synchronization is not supported.

  Note

  The data transmission service checks whether database or table aggregation exists only when a data migration or synchronization project is saved or started. The data transmission service does not block database or table aggregation that occurs during the running of a project. However, the mappings between databases or tables may fail to be correctly identified during reverse incremental synchronization, thereby compromising the data quality.

• At present, the data transmission service does not support the DDL statement CREATE DATABASE. If a new database created at the source meets the object matching rules, you must manually create a corresponding database at the destination to continue with the data synchronization of the new database.

Considerations

• After you configure object matching rules and exclusion rules, objects that are within the difference set of the object matching and exclusion rules can be selected.

  Note

  A difference set between two sets contains all elements that exist in one set but do not exist in the other set.

• After DDL synchronization is enabled, when you use a DDL statement to create a new table or modify the schema of a table at the source, if the table is within the difference set of the object matching and exclusion rules, this DDL statement can be synchronized to the destination by the data transmission service.

• Take note of the following items in a database or table aggregation scenario:

  • We recommend that you import objects or configure object matching rules to map objects between the source and the destination.

  • We recommend that you manually create schemas at the destination. If you create a schema by using the data transmission service, skip the failed objects in the schema migration step.

  • If you select DDL Synchronization, databases or tables may be deleted by mistake. For example, when multiple databases or tables at the source are aggregated to a single database or table at the destination, if a database or table is deleted at the source, the aggregated database or table may be deleted at the destination.

  • When you create a data migration or synchronization project, select Ignore for Processing Strategy When Records Exist in Target Object.

    Note

    If you select Ignore, data is pulled by using the IN mode during full verification. In this case, verification is inapplicable if the destination has a large amount of data, and the verification performance is downgraded.

• If a renaming mapping rule is configured for tables, the renaming mapping rule takes precedence. For example, if the rules a.b[0-3] and a.b[3-5]=a.c are configured, the a.b3 table is renamed as a.c.

• When you execute the DDL statement RENAME TABLE, if the renamed table does not meet the original matching or exclusion rule, unexpected synchronization issues may occur. Proceed with caution.

Procedure

1. Complete the steps before Select Migration Objects or Select Synchronization Objects for a data migration or synchronization project.

   For more information, see the documentation of data migration or data synchronization projects of the corresponding type.

   Note

   • The procedure for selecting migration objects is the same as that for selecting synchronization objects. This section describes how to select migration objects.

   • In a data synchronization project, you can configure wildcard-based matching rules to select synchronization objects only when you synchronize data from a MySQL tenant of OceanBase Database to an AnalyticDB for MySQL database, synchronize data from an ApsaraDB RDS for PostgreSQL instance to an Oracle tenant of OceanBase Database, or synchronize data between MySQL tenants of OceanBase Database.

2. On the Select Migration Objects page, select the migration objects for the current data migration project.

   You can use Specify Objects or Match Rules to select the migration objects. This section describes how to configure a matching rule.

   1. Select Match Rules.

   2. In the Specify Migration Scope section, specify object matching rules in the Object Migration Rule field and object exclusion rules in the Object Exclusion Rule field. The Object Exclusion Rule field is optional. For more information, see Wildcard patterns supported for matching rules.

   3. Click Verify.

      To view the matching results, click Preview Objects after the verification succeeds. The wildcard-based matching rules and exclusion rules apply to tables and views. The matching results are displayed on the Final Objects, New Objects, and Removed Objects tabs.

      Tab	Description
      Final Objects	Displays the final migration objects that match the configured matching rules.
      New Objects	Displays the new migration objects, which is a difference set of the final objects and the objects that meet the previous matching rules.
      Removed Objects	Displays the removed objects, which is a difference set of the final migration objects and the objects that meet the previous matching rules.

3. Complete subsequent project settings as prompted.

Sample scenarios

FAQ

• What can I do if the privileges are insufficient?

  Pay attention to the privilege settings of the source database user. If you do not grant all required privileges to the migration user, some objects are not displayed in the frontend by the data transmission service, and you cannot correctly configure matching rules. In this case, you must add these objects to Object Exclusion Rule to prevent the data migration or synchronization project from being interrupted because the data transmission service cannot find the target objects.

• What can I do if DML filtering is unsupported?

  If DDL synchronization is disabled, the data transmission service allows you to select objects based on matching rules. If a new table created during incremental synchronization meets a matching rule, the related DDL statements will be ignored but the data transmission service will continue to synchronize DML statements. As a result, the data migration or synchronization project will be interrupted because objects cannot be migrated or synchronized to the destination. Therefore, you must create the table at the destination or add the table to the blocklist.