Build a Data Sync Job via CreateDIJob API - DataWorks - Alibaba Cloud - DataWorks

Create a new version of a data integration task.

Operation description

This API requires DataWorks Basic Edition or a higher edition.
This API creates a data integration synchronization task. Key parameters include SourceDataSourceSettings, DestinationDataSourceSettings, and MigrationType. The TransformationRules parameter defines transformation rules for synchronized tables, such as adding columns or replacing table names. The TableMappings parameter specifies which tables to synchronize and their mapping rules. The JobSettings parameter configures task settings, including column mapping and scheduling.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

No authorization for this operation. If you encounter issues with this operation, contact technical support.

Request parameters

Parameter	Type	Required	Description	Example
DestinationDataSourceType	string	No	The type of the destination data source. Valid values: `Hologres`, `OSS-HDFS`, `OSS`, `MaxCompute`, `LogHub`, `StarRocks`, `DataHub`, `AnalyticDB for MySQL`, `Kafka`, and `Hive`.	Hologres
Description	string	No	The description of the job.	DI Job Demo
SourceDataSourceType	string	No	The type of the source data source. Valid values: `PolarDB`, `MySQL`, `Kafka`, `LogHub`, `Hologres`, `Oracle`, `OceanBase`, `MongoDB`, `Redshift`, `Hive`, `SQL Server`, `Doris`, and `ClickHouse`.	MySQL
ProjectId	integer	No	The ID of the DataWorks workspace for this API call. To obtain the workspace ID, log on to the DataWorks console and go to the Workspace Management page.	10000
Name	string	No	The name of the job.	mysql_to_holo_sync_8772
MigrationType	string	No	The synchronization type. Valid values: `FullAndRealtimeIncremental`: Full and real-time incremental synchronization for an entire database. `RealtimeIncremental`: Real-time incremental synchronization for a single table. `Full`: Full batch synchronization for an entire database. `OfflineIncremental`: Incremental synchronization in batch mode. `FullAndOfflineIncremental`: Full and incremental batch synchronization for an entire database.	FullAndRealtimeIncremental
JobType	string	No	The job type. Valid values: `DatabaseRealtimeMigration`: Synchronizes multiple tables from multiple source databases in real time (stream synchronization). This type supports full, incremental, or both full and incremental synchronization. `DatabaseOfflineMigration`: Synchronizes multiple tables from multiple source databases in batches. This type supports full, incremental, or both full and incremental synchronization. `SingleTableRealtimeMigration`: Synchronizes a single source table in real time (stream synchronization).	DatabaseRealtimeMigration
SourceDataSourceSettings	array<object>	No	Settings for the source data sources.
	array<object>	No	The settings for a single source data source.
DataSourceName	string	No	The name of the data source.	mysql_datasource_1
DataSourceProperties	object	No	The properties of the data source.
Encoding	string	No	The database encoding format.	UTF-8
Timezone	string	No	The time zone.	Asia/Shanghai
ConnectionProperties	string	No	Custom connection settings for the data source, such as instance ID, access credentials, and instance region. You must specify this parameter or `DataSourceName`. This parameter applies only when the data source is configured in instance mode (`ConnectionPropertiesMode`). The property format varies by data source. For more information, see ConnectionProperties for data sources.	{ "instanceId": "rm-2ze09gn3x6xxx", "password": "xxxx", "database": "agent", "username": "zmtest" "regionId": "cn-beijing" }
DestinationDataSourceSettings	array<object>	No	Settings for the destination data sources.
	array<object>	No	The settings for a single destination data source.
DataSourceName	string	No	The name of the data source.	holo_datasource_1
DataSourceProperties	object	No	The properties of the data source.
ConnectionProperties	string	No	Custom connection settings for the data source, such as instance ID, access credentials, and instance region. You must specify this parameter or `DataSourceName`. This parameter applies only when the data source is configured in instance mode (`ConnectionPropertiesMode`). The property format varies by data source. For more information, see ConnectionProperties for data sources.	{ "instanceId": "rm-2ze09gn3x6xxx", "password": "xxxx", "database": "agent", "username": "zmtest" "regionId": "cn-beijing" }
ResourceSettings	object	No	The resource settings.
OfflineResourceSettings	object	No	Resource settings for batch synchronization.
RequestedCu	number	No	The number of CUs for the resource group for data integration that is used for batch synchronization.	2
ResourceGroupIdentifier	string	No	The identifier of the resource group for data integration used for batch synchronization.	S_res_group_111_222
RealtimeResourceSettings	object	No	The resources for real-time synchronization.
RequestedCu	number	No	The number of CUs for the resource group for data integration that is used for real-time synchronization.	2
ResourceGroupIdentifier	string	No	The identifier of the resource group for data integration used for real-time synchronization.	S_res_group_111_222
ScheduleResourceSettings	object	No	The scheduling resources.
RequestedCu	number	No	The number of CUs for the scheduling resource group that is used for batch synchronization jobs.	2
ResourceGroupIdentifier	string	No	The identifier of the scheduling resource group used for batch synchronization jobs.	S_res_group_222_333
TransformationRules	array<object>	No	A list of transformation rules for the objects to be synchronized. Note [ { "RuleName":"my_database_rename_rule", "RuleActionType":"Rename", "RuleTargetType":"Schema", "RuleExpression":"{\"expression\":\"${srcDatasoureName}_${srcDatabaseName}\"}" } ]
	object	No	A single transformation rule.
RuleActionType	string	No	The action type. Valid values: `DefinePrimaryKey`: Defines a primary key. `Rename`: Renames an object. `AddColumn`: Adds a column. `HandleDml`: Handles DML operations. `DefineIncrementalCondition`: Defines an incremental condition. `DefineCycleScheduleSettings`: Defines periodic scheduling settings. `DefinePartitionKey`: Defines a partition key.	Rename
RuleExpression	string	No	The rule expression, specified as a JSON string. Renaming rule (`Rename`) Example: `{"expression":"${srcDatasourceName}_${srcDatabaseName}_0922" }` `expression`: The renaming expression. You can use the following variables: `${srcDatasourceName}` (name of the source data source), `${srcDatabaseName}` (name of the source database), and `${srcTableName}` (name of the source table). Rule for adding a column (`AddColumn`) Example: `{"columns":[{"columnName":"my_add_column","columnValueType":"Constant","columnValue":"123"}]}` If you do not specify this rule, no columns are added. `columnName`: The name of the column to add. `columnValueType`: The value type of the added column. Valid values: `Constant` and `Variable`. `columnValue`: The value of the added column. If `columnValueType` is `Constant`, the value is a custom string constant. If `columnValueType` is `Variable`, the value is a built-in variable. Valid built-in variables include: `EXECUTE_TIME` (execution time, Long), `DB_NAME_SRC` (source database name, String), `DATASOURCE_NAME_SRC` (source data source name, String), `TABLE_NAME_SRC` (source table name, String), `DB_NAME_DEST` (destination database name, String), `DATASOURCE_NAME_DEST` (destination data source name, String), `TABLE_NAME_DEST` (destination table name, String), and `DB_NAME_SRC_TRANSED` (transformed database name, String). Rule for defining the primary key columns of a destination table (`DefinePrimaryKey`) Example: `{"columns":["ukcolumn1","ukcolumn2"]}` If you do not specify this rule, the primary key columns of the source table are used by default. If the destination is an existing table, Data Integration does not modify its schema. If a specified primary key column does not exist in the destination table, the job fails to start and an error is reported. If the destination table is automatically created, Data Integration automatically creates its schema with the defined primary key columns. If a specified primary key column does not exist in the source table, the job fails to start and an error is reported. DML handling rule (`HandleDml`) Example: `{"dmlPolicies":[{"dmlType":"Delete","dmlAction":"Filter","filterCondition":"id > 1"}]}` If you do not specify this rule, the default `dmlAction` is `Normal` for `Insert`, `Update`, and `Delete` operations. `dmlType`: The DML operation type. Valid values: `Insert`, `Update`, and `Delete`. `dmlAction`: The DML handling policy. Valid values: `Normal` (normal processing), `Ignore`, `Filter` (conditional processing, used when `dmlType` is `Update` or `Delete`), and `LogicalDelete` (logical deletion). `filterCondition`: The DML filter condition, used when `dmlAction` is `Filter`. Incremental condition (`DefineIncrementalCondition`) Example: `{"where":"id > 0"}` Specifies the filter condition for incremental synchronization. Parameters for periodic scheduling (`DefineCycleScheduleSettings`) Example: `{"cronExpress":" * * * * * *", "cycleType":"1"}` Specifies the parameters for periodically scheduling a job. Rule to define a partition key (`DefinePartitionKey`) Example: `{"columns":["id"]}` Specifies a partition key.	{ "expression": "${srcDatasoureName}_${srcDatabaseName}" }
RuleName	string	No	The name of the rule. The rule name must be unique for a specific action type and target object type. Maximum length: 50 characters.	rename_rule_1
RuleTargetType	string	No	The type of the object to which the action applies. Valid values: `Table` `Schema` `Database`	Table
TableMappings	array<object>	No	Transformation mappings for the objects to be synchronized. Each mapping defines selection rules for a group of source objects and the transformation rules to apply to them. Note [ { "SourceObjectSelectionRules":[ { "ObjectType":"Database", "Action":"Include", "ExpressionType":"Exact", "Expression":"biz_db" }, { "ObjectType":"Schema", "Action":"Include", "ExpressionType":"Exact", "Expression":"s1" }, { "ObjectType":"Table", "Action":"Include", "ExpressionType":"Exact", "Expression":"table1" } ], "TransformationRuleNames":[ { "RuleName":"my_database_rename_rule", "RuleActionType":"Rename", "RuleTargetType":"Schema" } ] } ]
	array<object>	No	A mapping that selects a set of source objects and specifies the transformation rules to apply.
SourceObjectSelectionRules	array<object>	No	Each rule can select a set of source objects to synchronize. Multiple rules combine to select one table.
	object	No	A rule that selects source objects of a specific type, such as a database or table.
Action	string	No	The selection action. Valid values: `Include` and `Exclude`.	Include
Expression	string	No	The expression.	mysql_table_1
ExpressionType	string	No	The expression type. Valid values: `Exact` and `Regex`.	Exact
ObjectType	string	No	The object type. Valid values: `Table` `Schema` `Database`	Table
TransformationRules	array<object>	No	The names of the transformation rules to apply to the selected objects.
	object	No	The transformation rule applied to the source object.
RuleName	string	No	The name of the transformation rule. The rule name must be unique for a specific action type and target object type. Maximum length: 50 characters.	rename_rule_1
RuleActionType	string	No	The action type. Valid values: `DefinePrimaryKey`: Defines a primary key. `Rename`: Renames an object. `AddColumn`: Adds a column. `HandleDml`: Handles DML operations. `DefineIncrementalCondition`: Defines an incremental condition. `DefineCycleScheduleSettings`: Defines periodic scheduling settings. `DefinePartitionKey`: Defines a partition key.	Rename
RuleTargetType	string	No	The type of the object to which the action applies. Valid values: `Table` `Schema` `Database`	Table
JobSettings	object	No	The settings for the synchronization job, including DDL processing policies, data type mappings between source and destination columns, and runtime parameters.
ChannelSettings	string	No	Settings for data synchronization channels. You can configure special settings for specific channels. The following channels are supported: synchronization from Hologres to Hologres (Holo2Holo) and from Hologres to Kafka (Holo2Kafka). Holo2Kafka Example: `{"destinationChannelSettings":{"kafkaClientProperties":[{"key":"linger.ms","value":"100"}],"keyColumns":["col3"],"writeMode":"canal"}}` `kafkaClientProperties`: Parameters for the Kafka producer. `keyColumns`: The columns whose values are used as keys for data written to Kafka. `writeMode`: The data format for writing to Kafka. Valid values: `json` and `canal`. Holo2Holo Example: `{"destinationChannelSettings":{"conflictMode":"replace","dynamicColumnAction":"replay","writeMode":"replay"}}` `conflictMode`: The conflict handling policy for writing data to Hologres. Valid values: `replace` (overwrite) and `ignore`. `writeMode`: The method for writing data to Hologres. Valid values: `replay` and `insert`. `dynamicColumnAction`: The method for handling dynamic columns when writing data to Hologres. Valid values: `replay`, `insert`, and `ignore`.	{ "structInfo": "MANAGED", "storageType": "TEXTFILE", "writeMode": "APPEND", "partitionColumns": [ { "columnName": "pt", "columnType": "STRING", "comment": "" } ], "fieldDelimiter": "" }
ColumnDataTypeSettings	array<object>	No	Column data type mappings. Note "ColumnDataTypeSettings":[ { "SourceDataType":"Bigint", "DestinationDataType":"Text" } ]
	object	No	A mapping from a source data type to a destination data type.
DestinationDataType	string	No	The destination data type. For example: `bigint`, `boolean`, `string`, `text`, `datetime`, `timestamp`, `decimal`, or `binary`. Available data types vary by data source.	text
SourceDataType	string	No	The source data type. For example: `bigint`, `boolean`, `string`, `text`, `datetime`, `timestamp`, `decimal`, or `binary`. Available data types vary by data source.	bigint
CycleScheduleSettings	object	No	The periodic scheduling settings.
CycleMigrationType	string	No	The synchronization type for periodic scheduling. Valid values: `Full`: Full synchronization. `OfflineIncremental`: Incremental synchronization in batch mode.	Full
ScheduleParameters	string	No	The scheduling parameters.	bizdate=$bizdate
DdlHandlingSettings	array<object>	No	DDL handling settings. Note "DDLHandlingSettings":[ { "Type":"Insert", "Action":"Normal" } ]
	object	No	A single DDL handling setting.
Action	string	No	The handling action. Valid values: `Ignore`: Ignores the DDL message. `Critical`: Reports an error. `Normal`: Processes the DDL message normally.	Critical
Type	string	No	The DDL type. Valid values: `RenameColumn`: Renames a column. `ModifyColumn`: Modifies a column. `CreateTable`: Creates a table. `TruncateTable`: Truncates a table. `DropTable`: Drops a table. `DropColumn`: Drops a column. `AddColumn`: Adds a column.	AddColumn
RuntimeSettings	array<object>	No	The runtime settings.
	object	No	A runtime setting.
Name	string	No	The name of the setting. Valid values: `src.offline.datasource.max.connection`: The maximum number of connections to the source of a batch synchronization job. `dst.offline.truncate`: Specifies whether to truncate the destination table before a batch job starts. `runtime.offline.speed.limit.enable`: Specifies whether to enable throttling for a batch synchronization job. `runtime.offline.concurrent`: The concurrency level of a batch synchronization job. `runtime.enable.auto.create.schema`: Specifies whether to automatically create a destination schema. `runtime.realtime.concurrent`: The concurrency level of a real-time synchronization job. `runtime.realtime.failover.minute.dataxcdc`: The wait time in minutes for a failover restart. `runtime.realtime.failover.times.dataxcdc`: The number of failover restart attempts.	runtime.offline.concurrent
Value	string	No	The value of the setting.	1
JobName `deprecated`	string	No	This parameter is deprecated. Use the `Name` parameter instead.	mysql_to_holo_sync_8772
Owner	string	No	The job owner.	3726346
FileSpec	string	No	The code for a job created in script mode.	{ "resourceSettings": { "realtimeResourceSettings": { "requestedCu": 2, "resourceGroupIdentifier": "Serverless_res_group_123_456" }, "offlineResourceSettings": { "requestedCu": 2, "resourceGroupIdentifier": "Serverless_res_group_123_456" } }, "tableMappings": [ { "sourceObjectSelectionRules": [ { "expression": "autotest_hologres", "action": "Include", "expressionType": "Exact", "objectType": "Datasource" }, { "expression": "auto_holo_2661647", "action": "Include", "expressionType": "Exact", "objectType": "Table" }, { "expression": "public", "action": "Include", "expressionType": "Exact", "objectType": "Schema" } ], "transformationRules": [ { "ruleTargetType": "Table", "ruleActionType": "SourceSchema", "ruleName": "SourceSchema_Table_BStf8aXPSCJjOWGe" }, { "ruleTargetType": "Schema", "ruleActionType": "Rename", "ruleName": "Rename_Schema_3qWNOIsljtInvKJy" }, { "ruleTargetType": "Table", "ruleActionType": "Rename", "ruleName": "Rename_Table_o3PVQq1aIKDGoVVW" }, { "ruleTargetType": "Table", "ruleActionType": "DefineDstTableSettings", "ruleName": "DefineDstTableSettings_Table_BhJltOmOCIc81fzi" }, { "ruleTargetType": "Table", "ruleActionType": "ColumnMapping", "ruleName": "ColumnMapping_Table_nP4hJPX1wh2W3fpo" } ] } ], "sourceDataSourceSettings": [ { "dataSourceProperties": { "timeZone": "Asia/Shanghai" }, "dataSourceName": "autotest_hologres" } ], "jobSettings": { "runtimeSettings": [ ], "ddlHandlingSettings": [ ], "columnDataTypeSettings": [ ], "cycleScheduleSettings": { }, "channelSettings": { "destinationChannelSettings": { "conflictMode": "replace", "dynamicColumnAction": "replay", "writeMode": "replay" }, "sourceChannelSettings": { } } }, "destinationDataSourceType": "Hologres", "transformationRules": [ { "ruleTargetType": "Table", "ruleName": "SourceSchema_Table_BStf8aXPSCJjOWGe", "ruleActionType": "SourceSchema", "ruleExpression": { "columns": [ { "name": "id", "category": "normal", "type": "BIGINT" }, { "name": "decimal", "category": "normal", "type": "DECIMAL" } ] } }, { "ruleTargetType": "Schema", "ruleName": "Rename_Schema_3qWNOIsljtInvKJy", "ruleActionType": "Rename", "ruleExpression": { "expression": "public" } }, { "ruleTargetType": "Table", "ruleName": "Rename_Table_o3PVQq1aIKDGoVVW", "ruleActionType": "Rename", "ruleExpression": { "expression": "auto_holo_2661647_dst" } }, { "ruleTargetType": "Table", "ruleName": "DefineDstTableSettings_Table_BhJltOmOCIc81fzi", "ruleActionType": "DefineDstTableSettings", "ruleExpression": { "ddlString": "BEGIN; CREATE TABLE IF NOT EXISTS public.auto_holo_2661647_dst ( id BIGINT PRIMARY KEY, "decimal" DECIMAL(38,18) ); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'time_to_live_in_seconds', '3153600000'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'orientation', 'column'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'binlog.level', 'replica'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'binlog.ttl', '2592000'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'bitmap_columns', '"text","char","varchar"'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'dictionary_encoding_columns', '"text":auto,"bytea":auto,"char":auto,"varchar":auto'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'distribution_key', '"id"'); COMMIT; ", "ddlType": "STRUCT" } }, { "ruleTargetType": "Table", "ruleName": "ColumnMapping_Table_nP4hJPX1wh2W3fpo", "ruleActionType": "ColumnMapping", "ruleExpression": { "columnMapping": [ { "sourceColName": "id", "dstColName": "id" }, { "sourceColName": "decimal", "dstColName": "decimal" } ] } } ], "migrationType": "FullAndRealtimeIncremental", "destinationDataSourceSettings": [ { "dataSourceProperties": { }, "dataSourceName": "autotest_hologres" } ], "sourceDataSourceType": "Hologres" }

Response elements

Element	Type	Description	Example
	object	The response schema.
Id	integer	The data integration job ID.	11792
RequestId	string	The request ID. Use this ID to locate logs and troubleshoot issues.	4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB
DIJobId `deprecated`	integer	This field is deprecated. Use the `Id` field instead.	11792

Examples

Success response

JSON format

{
  "Id": 11792,
  "RequestId": "4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB",
  "DIJobId": 11792
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.