When you call the API operations of Data Transmission Service (DTS), you can specify or query the objects of data migration, data synchronization, or change tracking tasks. This topic describes the related API operations and provides the object definitions and configuration examples.
Related API operations and parameters
API operation | Description |
---|---|
You can use the request parameter DbList to specify the objects of data migration, data synchronization, or change tracking
tasks.
|
|
You can use the response parameter DbObject to query the objects of data migration, data synchronization, or change tracking
tasks.
|
Object definitions
The value of an object-related parameter is a JSON string.
Note You can configure the italic content based on your needs.
- If the objects contain multiple databases, you can define the objects as shown in
the following example:
{ "The name of source database 1": { "name": "The database name that you want to use in the destination instance", "all": true indicates that the entire database is used as the source object }, "The name of source database 2": { "name": "The database name that you want to use in the destination instance", "all": false indicates that the source object is not the entire database, "Table": { "The name of source table A": { "name": "The table name that you want to use in the destination instance", "all": true indicates that the entire table is used as the source object, "dml_op": "The DML operations that you want to migrate or synchronize", "ddl_op": "The DDL operations that you want to migrate or synchronize" } } }, "The name of source database 3": { "name": "The database name that you want to use in the destination instance", "all": true indicates that the entire database is used as the source object, "dml_op": "The DML operations that you want to migrate or synchronize", "ddl_op": "The DDL operations that you want to migrate or synchronize" } }
- If the objects are specific columns or include filter conditions, you can define the
objects as shown in the following example:
{ "The name of the source database for the data migration, data synchronization, or change tracking task": { "name": "The database name that you want to use in the destination instance", "all": false indicates that the source object is not the entire database, "Table": { "The name of source table A": { "name": "The table name that you want to use in the destination instance", "all": false indicates that the source object is not the entire table, "filter": "id>10" "column": { "id": { "key": "PRI", "name": "id", "type": "int(11)", "sharedKey": false, "state": "checked" } }, "shard": 12 } }, } }
- If you migrate or synchronize objects to an AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance, you can define the objects as shown in the following example:
{ "The name of the source database that you want to migrate or synchronize": { "name": "The database name that you want to use in the destination instance", "all": Valid value: false. If the destination instance is an AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance, you must set the value to false regardless of whether the entire database or specific tables are used as the source objects. In this case, you must also specify the partition key corresponding to each table. "Table": { "The name of source table A": { "all": true indicates that the entire table is used as the source object, "name": "The table name that you want to use in the destination instance", "primary_key": "The primary key, for example, id", "type": "The type of the table, for example, dimension", } "The name of source table B": { "all": true indicates that the entire table is used as the source object, "name": "The table name that you want to use in the destination instance", "part_key": "The partition key, for example, id", "primary_key": "The primary key, for example, id", "type": "The type of the table, for example, partition", } } } }
Parameter | Description |
---|---|
name |
The database name, table name, or column name that you want to use in the destination instance. For example, if you want to migrate a database named dtssource to a database named dtstarget, you must set the name parameter to dtstarget. |
all |
Specifies whether to select all tables or columns. Valid values:
|
Table |
The information of the source table. |
filter |
The condition used to filter the data to be migrated or synchronized. The filter condition
takes effect only at the table level.
For example, you can enter
id>10 to migrate or synchronize only the data records whose id column value is greater
than 10. For more information about the format of the filter condition, see Use SQL conditions to filter data.
Note You cannot set filter conditions for change tracking tasks.
|
column |
The information of the source column. |
key |
Specifies whether to use the column as the primary key. Valid values:
|
sharedKey |
Specifies whether to use the column as the shard key. Valid values:
Note This parameter is required only if the source database is Kafka.
|
type |
The data type of the field. |
state |
The checked state indicates that the column is selected.
|
shard |
The number of shards in the source table.
Note This parameter is required only if the source database is Kafka.
|
dml_op |
The DML operations that you want to migrate or synchronize. Valid values:
Note For more information about the DML operations supported by different data migration
or synchronization tasks, see the topics listed in Overview of data migration scenarios or Overview of data synchronization scenarios.
|
ddl_op |
The DDL operations that you want to migrate or synchronize. Valid values:
Note For more information about the DDL operations supported by different data migration
or synchronization tasks, see the topics listed in Overview of data migration scenarios or Overview of data synchronization scenarios.
|
primary_key |
The primary key. This parameter is available and must be specified only if the destination instance is an AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance. |
part_key |
The partition key. This parameter is available and must be specified only if the destination instance is an AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance. |
type Notice The
type parameter here is not the same as the type parameter that represents the data type of a field.
|
If the destination instance is an AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance, you must specify the type of each source table. Valid values:
|
Configuration examples
- Example 1: Replicate data from all tables in the dtstestdata database.
{"dtstestdata": { "name": "dtstestdata", "all": true }}
- Example 2: Replicate data from all tables in the dtstestdata database and change the
database name to dtstestdata_new.
{"dtstestdata": { "name": "dtstestdata_new", "all": true }}
- Example 3: Replicate data from a table named customer in the dtstestdata database.
{"dtstestdata": { "name": "dtstestdata", "all": false, "Table": { "customer": { "name": "customer", "all": true, "column": { "id": { "key": "PRI", "name": "id", "type": "int(11)", "sharedKey": false, "state": "checked" }, "gmt_create": { "key": "", "name": "gmt_create", "type": "datetime", "sharedKey": false, "state": "checked" }, "gmt_modify": { "key": "", "name": "gmt_modify", "type": "datetime", "sharedKey": false, "state": "checked" }, "valid_time": { "key": "", "name": "valid_time", "type": "datetime", "sharedKey": false, "state": "checked" }, "creator": { "key": "", "name": "creator", "type": "varchar(200)", "sharedKey": false, "state": "checked" } }, "shard": 12 } }, } }
- Example 4: Replicate data from specific columns of the customer and order tables in
the dtstestdata database.
{"dtstestdata": { "name": "dtstestdata", "all": false, "Table": { "customer": { "name": "customer", "all": false, "column": { "id": { "key": "PRI", "name": "id", "type": "int(11)", "sharedKey": false, "state": "checked" }, "level": { "key": "", "name": "level", "type": "varchar(5000)", "sharedKey": false, "state": "checked" }, "name": { "key": "", "name": "name", "type": "varchar(500)", "sharedKey": false, "state": "checked" }, }, "shard": 12 }, "order": { "name": "order", "all": false, "column": { "id": { "key": "PRI", "name": "id", "type": "int(11)", "sharedKey": false, "state": "checked" } }, "shard": 12 } }, } }
- Example 5: Replicate data from the customer, order, and commodity tables in the dtstestdata
database to the destination AnalyticDB for MySQL or AnalyticDB for PostgreSQL instance.
{ "dtstestdata": { "name": "dtstestdatanew", "all": false, "Table": { "order": { "name": "ordernew", "all": true, "part_key": "id", "primary_key": "id", "type": "partition" }, "customer": { "name": "customernew", "all": true, "primary_key": "id", "type": "dimension" }, "commodity": { "name": "commoditynew", "all": false, "filter": "id>10", "column": { "id": { "key": "PRI", "name": "id", "type": "int(11)" } }, "part_key": "id", "primary_key": "id", "type": "partition" } } } }
Supported regular expressions
Symbol | Description |
---|---|
Period (.) | Matches any single character except '\r\n'. |
Asterisk (*) | Matches zero or more repetitions of the preceding sub-expression. For example, h.*llo matches strings such as hllo and heeeello.
|
Question mark (?) | Matches zero or one repetition of the preceding sub-expression. For example, h.?llo matches hllo and hello, but not haello.
|
Character set [characters] | Matches any single character included in the brackets. For example, h[ae]llo matches hallo and hello.
|
Negative character set [^characters] | Matches any single character except the characters included in the brackets. For example,
h[ae]llo matches hcllo or hdllo, but not hallo or hello.
|
Character range [character1-character2] | Matches any characters within the range from character1 to character2, for example, [0-9] and [a-z]. |