CreateDIJob - Create data integration task | DataWorks API - DataWorks

Creates a data integration task.

Operation description

You must purchase DataWorks Basic Edition or a higher edition to use this feature.
This operation creates a data integration synchronization task. Parameters include the source configuration SourceDataSourceSettings and the destination configuration DestinationDataSourceSettings, the supported synchronization type MigrationType, transformation rules defined through TransformationRules for mapping operations such as adding columns and renaming tables, specific tables to synchronize and the mapping rules to apply defined in TableMappings, and task-level settings such as column mappings and scheduling configurations defined in JobSettings.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

No authorization for this operation. If you encounter issues with this operation, contact technical support.

Request parameters

Parameter	Type	Required	Description	Example
DestinationDataSourceType	string	No	The type of the destination data source. Valid values: Hologres, OSS-HDFS, OSS, MaxCompute, LogHub, StarRocks, DataHub, AnalyticDB_For_MySQL, Kafka, Hive.	Hologres
Description	string	No	The description of the task.	DI Job Demo
SourceDataSourceType	string	No	The type of the source data source. Valid values: PolarDB, MySQL, Kafka, LogHub, Hologres, Oracle, OceanBase, MongoDB, RedShift, Hive, SQLServer, Doris, ClickHouse.	MySQL
ProjectId	integer	No	The ID of the DataWorks workspace. You can log on to the DataWorks console and go to the workspace management page to obtain the ID. This parameter specifies the DataWorks workspace for this API call.	10000
Name	string	No	The name of the task.	mysql_to_holo_sync_8772
MigrationType	string	No	The synchronization type. Valid values: FullAndRealtimeIncremental: full and real-time incremental synchronization for entire databases in real time. RealtimeIncremental: real-time incremental synchronization for single tables in real time. Full: full synchronization for entire databases offline. OfflineIncremental: offline incremental synchronization for entire databases offline. FullAndOfflineIncremental: full and offline incremental synchronization for entire databases offline.	FullAndRealtimeIncremental
JobType	string	No	The task type. Valid values: DatabaseRealtimeMigration: real-time migration of entire databases. Performs streaming synchronization of multiple tables from multiple source databases. Supports full-only, incremental-only, or full and incremental synchronization. DatabaseOfflineMigration: offline migration of entire databases. Performs batch synchronization of multiple tables from multiple source databases. Supports full-only, incremental-only, or full and incremental synchronization. SingleTableRealtimeMigration: real-time migration of a single table. Performs streaming synchronization of a single source table.	DatabaseRealtimeMigration
SourceDataSourceSettings	array<object>	No	The list of source data source settings.
	array<object>	No	A single source data source setting.
DataSourceName	string	No	The name of the data source.	mysql_datasource_1
DataSourceProperties	object	No	The properties of the data source.
Encoding	string	No	The database encoding.	UTF-8
Timezone	string	No	The time zone.	Asia/Shanghai
ConnectionProperties	string	No	Specify either this parameter or DataSourceName. This parameter specifies custom data source connection configuration information, including the instance ID, access identity, and instance region. This parameter supports only datasource config in instance pattern (ConnectionPropertiesMode). Different data sources have different property specifications. For more information, see Data source connection information ConnectionProperties.	{ "instanceId": "rm-2ze09gn3x6xxx", "password": "xxxx", "database": "agent", "username": "zmtest" "regionId": "cn-beijing" }
DestinationDataSourceSettings	array<object>	No	The list of destination data source settings.
	array<object>	No	A single destination data source setting.
DataSourceName	string	No	The name of the data source.	holo_datasource_1
DataSourceProperties	object	No	The properties of the data source.
ConnectionProperties	string	No	Specify either this parameter or DataSourceName. This parameter specifies custom data source connection configuration information, including the instance ID, access identity, and instance region. This parameter supports only datasource config in instance pattern (ConnectionPropertiesMode). Different data sources have different property specifications. For more information, see Data source connection information ConnectionProperties.	{ "instanceId": "rm-2ze09gn3x6xxx", "password": "xxxx", "database": "agent", "username": "zmtest" "regionId": "cn-beijing" }
ResourceSettings	object	No	The resource settings.
OfflineResourceSettings	object	No	The offline synchronization resource settings.
RequestedCu	number	No	The number of compute units (CUs) of the data integration resource group used for offline synchronization.	2
ResourceGroupIdentifier	string	No	The name of the data integration resource group used for offline synchronization.	S_res_group_111_222
RealtimeResourceSettings	object	No	The real-time synchronization resource settings.
RequestedCu	number	No	The number of CUs of the data integration resource group used for real-time synchronization.	2
ResourceGroupIdentifier	string	No	The name of the data integration resource group used for real-time synchronization.	S_res_group_111_222
ScheduleResourceSettings	object	No	The schedule resource settings.
RequestedCu	number	No	The number of CUs of the scheduling resource group used for offline synchronization nodes.	2
ResourceGroupIdentifier	string	No	The name of the scheduling resource group used for offline synchronization nodes.	S_res_group_222_333
TransformationRules	array<object>	No	The list of synchronization object transformation rule definitions. Note [ { "RuleName":"my_database_rename_rule", "RuleActionType":"Rename", "RuleTargetType":"Schema", "RuleExpression":"{"expression":"${srcDatasoureName}_${srcDatabaseName}"}" } ]
	object	No	A single synchronization object transformation rule definition. Each element represents one transformation rule.
RuleActionType	string	No	The action type. Valid values: DefinePrimaryKey: defines a primary key. Rename: renames an object. AddColumn: adds a column. HandleDml: handles DML operations. DefineIncrementalCondition: defines an incremental condition. DefineCycleScheduleSettings: defines cycle scheduling settings. DefinePartitionKey: defines a partition key.	Rename
RuleExpression	string	No	The rule expression in JSON string format. Rename rule (Rename) Example: {"expression":"${srcDatasourceName}_${srcDatabaseName}_0922" } expression: the rename transformation rule expression. The expression supports the following variables: ${srcDatasourceName} (source data source name), ${srcDatabaseName} (source database name), and ${srcTableName} (source table name). Add column rule (AddColumn) Example: {"columns":[{"columnName":"my_add_column","columnValueType":"Constant","columnValue":"123"}]} If not specified, the default rule is to neither add columns nor perform replication. columnName: the name of the additional column. columnValueType: the value type of the additional column. Valid values: Constant and Variable. columnValue: the value of the additional column. When columnValueType is set to Constant, the value is a custom constant of the String type. When columnValueType is set to Variable, the value is a built-in variable. Valid built-in variables: EXECUTE_TIME (execution time, Long type), DB_NAME_SRC (source database name, String type), DATASOURCE_NAME_SRC (source data source name, String type), TABLE_NAME_SRC (source table name, String type), DB_NAME_DEST (destination database name, String type), DATASOURCE_NAME_DEST (destination data source name, String type), TABLE_NAME_DEST (destination table name, String type), and DB_NAME_SRC_TRANSED (transformed database name, String type). Define primary key columns for the destination table (DefinePrimaryKey) Example: {"columns":["ukcolumn1","ukcolumn2"]} If not specified, the source primary key columns are used by default. When the destination table already exists: the data integration system does not modify the destination table schema. If the specified primary key columns are not in the destination column set, the node reports an error upon startup. When the destination table uses automatic creation: the data integration system automatically creates the destination table schema, which includes the defined primary key columns. If the specified primary key columns are not in the destination column set, the node reports an error upon startup. DML operations handling rule (HandleDml) Example: {"dmlPolicies":[{"dmlType":"Delete","dmlAction":"Filter","filterCondition":"id > 1"}]} If not specified, the default rule is Normal for Insert, Update, and Delete. dmlType: the DML operations type. Valid values: Insert, Update, and Delete. dmlAction: the DML operations handling policy. Valid values: Normal (process normally), Ignore (ignore), Filter (conditionally process normally, used when dmlType is Update or Delete), and LogicalDelete (logical delete). filterCondition: the DML filter condition, used when dmlAction is set to Filter. Incremental condition (DefineIncrementalCondition) Example: {"where":"id > 0"} Specifies the incremental filter condition. Cycle scheduling parameters (DefineCycleScheduleSettings) Example: {"cronExpress":" * * * * * *", "cycleType":"1"} Specifies the cycle node scheduling parameters. Define partition key (DefinePartitionKey) Example: {"columns":["id"]} Specifies the partition key.	{ "expression": "${srcDatasoureName}_${srcDatabaseName}" }
RuleName	string	No	The rule name. When the action type and target type are the same, the rule name must be unique. The name cannot exceed 50 characters in length.	rename_rule_1
RuleTargetType	string	No	The target type on which the action is applied. Valid values: Table Schema Database	Table
TableMappings	array<object>	No	The list of synchronization object transformation mappings. Each element describes a group of source object selection rules and the transformation rules applied to that group. Note [ { "SourceObjectSelectionRules":[ { "ObjectType":"Database", "Action":"Include", "ExpressionType":"Exact", "Expression":"biz_db" }, { "ObjectType":"Schema", "Action":"Include", "ExpressionType":"Exact", "Expression":"s1" }, { "ObjectType":"Table", "Action":"Include", "ExpressionType":"Exact", "Expression":"table1" } ], "TransformationRuleNames":[ { "RuleName":"my_database_rename_rule", "RuleActionType":"Rename", "RuleTargetType":"Schema" } ] } ]
	array<object>	No	Each rule represents the selection of a table to synchronize.
SourceObjectSelectionRules	array<object>	No	Each rule selects a set of source objects to synchronize. Multiple rules together select a single table.
	object	No	Each rule selects different object types of the source objects to synchronize, such as the source database or source table.
Action	string	No	The selection action. Valid values: Include and Exclude.	Include
Expression	string	No	The expression.	mysql_table_1
ExpressionType	string	No	The expression type. Valid values: Exact and Regex.	Exact
ObjectType	string	No	The object type. Valid values: Table Schema Database	Table
TransformationRules	array<object>	No	The list of synchronization object transformation rule definitions. Each element represents one transformation rule.
	object	No	The transformation rule applied to the source objects.
RuleName	string	No	The rule name. The name must be unique within the same action type and target type combination. The name cannot exceed 50 characters in length.	rename_rule_1
RuleActionType	string	No	The action type. Valid values: DefinePrimaryKey: defines a primary key. Rename: renames an object. AddColumn: adds a column. HandleDml: handles DML operations. DefineIncrementalCondition: defines an incremental condition. DefineCycleScheduleSettings: defines cycle scheduling settings. DefinePartitionKey: defines a partition key.	Rename
RuleTargetType	string	No	The target type on which the action is applied. Valid values: Table Schema Database	Table
JobSettings	object	No	The task-level settings, including DDL handling policies, source-to-destination column data type mapping policies, and task runtime parameters.
ChannelSettings	string	No	The channel-related task settings. You can configure special settings for specific channels. Currently supported channels include Holo2Holo (synchronization from Hologres to Hologres) and Holo2Kafka (synchronization from Hologres to Kafka). Holo2Kafka Example: {"destinationChannelSettings":{"kafkaClientProperties":[{"key":"linger.ms","value":"100"}],"keyColumns":["col3"],"writeMode":"canal"}} kafkaClientProperties: Kafka producer parameters used when writing to Kafka. keyColumns: the columns whose values are written to Kafka. writeMode: the Kafka write format. Currently supports json and canal. Holo2Holo Example: {"destinationChannelSettings":{"conflictMode":"replace","dynamicColumnAction":"replay","writeMode":"replay"}} conflictMode: the conflict handling policy when writing to Hologres. Valid values: replace (overwrite) and ignore. writeMode: the write mode for Hologres. Valid values: replay and insert. dynamicColumnAction: the dynamic column handling mode when writing to Hologres. Valid values: replay, insert, and ignore.	{ "structInfo": "MANAGED", "storageType": "TEXTFILE", "writeMode": "APPEND", "partitionColumns": [ { "columnName": "pt", "columnType": "STRING", "comment": "" } ], "fieldDelimiter": "" }
ColumnDataTypeSettings	array<object>	No	The column data type mapping array. Note ["ColumnDataTypeSettings":[ { "SourceDataType":"Bigint", "DestinationDataType":"Text" } ]
	object	No	A single column data type mapping.
DestinationDataType	string	No	The destination data type, such as bigint, boolean, string, text, datetime, timestamp, decimal, or binary. The available types vary by data source type.	text
SourceDataType	string	No	The source data type, such as bigint, boolean, string, text, datetime, timestamp, decimal, or binary. The available types vary by data source type.	bigint
CycleScheduleSettings	object	No	The cycle scheduling settings.
CycleMigrationType	string	No	The synchronization type that requires cycle scheduling. Valid values: Full: full synchronization. OfflineIncremental: offline incremental synchronization.	Full
ScheduleParameters	string	No	The scheduling parameters.	bizdate=$bizdate
DdlHandlingSettings	array<object>	No	The DDL handling settings array. Note ["DDLHandlingSettings":[ { "Type":"Insert", "Action":"Normal" } ]
	object	No	A single DDL handling setting.
Action	string	No	The handling action. Valid values: Ignore: ignores the DDL operation. Critical: reports an error. Normal: processes the DDL operation normally.	Critical
Type	string	No	The DDL type. Valid values: RenameColumn: renames a column. ModifyColumn: modifies a column. CreateTable: creates a table. TruncateTable: truncates a table. DropTable: drops a table. DropColumn: drops a column. AddColumn: adds a column.	AddColumn
RuntimeSettings	array<object>	No	The runtime settings.
	object	No
Name	string	No	The setting name. Valid values: src.offline.datasource.max.connection: the maximum number of connections to the source for offline batch tasks. dst.offline.truncate: specifies whether to truncate the destination table. runtime.offline.speed.limit.enable: specifies whether to enable throttling for offline batch tasks. runtime.offline.concurrent: the concurrency of offline batch synchronization tasks. runtime.enable.auto.create.schema: specifies whether to automatically create a schema on the destination. runtime.realtime.concurrent: the concurrency of real-time tasks. runtime.realtime.failover.minute.dataxcdc: the wait time in minutes before restarting after a failover failure. runtime.realtime.failover.times.dataxcdc: the number of restart attempts after a failover failure.	runtime.offline.concurrent
Value	string	No	The setting value.	1
JobName `deprecated`	string	No	[Deprecated] Use the Name parameter instead.	mysql_to_holo_sync_8772
Owner	string	No	The owner of the task.	3726346
FileSpec	string	No	The code content in script mode.	{ "resourceSettings": { "realtimeResourceSettings": { "requestedCu": 2, "resourceGroupIdentifier": "Serverless_res_group_123_456" }, "offlineResourceSettings": { "requestedCu": 2, "resourceGroupIdentifier": "Serverless_res_group_123_456" } }, "tableMappings": [ { "sourceObjectSelectionRules": [ { "expression": "autotest_hologres", "action": "Include", "expressionType": "Exact", "objectType": "Datasource" }, { "expression": "auto_holo_2661647", "action": "Include", "expressionType": "Exact", "objectType": "Table" }, { "expression": "public", "action": "Include", "expressionType": "Exact", "objectType": "Schema" } ], "transformationRules": [ { "ruleTargetType": "Table", "ruleActionType": "SourceSchema", "ruleName": "SourceSchema_Table_BStf8aXPSCJjOWGe" }, { "ruleTargetType": "Schema", "ruleActionType": "Rename", "ruleName": "Rename_Schema_3qWNOIsljtInvKJy" }, { "ruleTargetType": "Table", "ruleActionType": "Rename", "ruleName": "Rename_Table_o3PVQq1aIKDGoVVW" }, { "ruleTargetType": "Table", "ruleActionType": "DefineDstTableSettings", "ruleName": "DefineDstTableSettings_Table_BhJltOmOCIc81fzi" }, { "ruleTargetType": "Table", "ruleActionType": "ColumnMapping", "ruleName": "ColumnMapping_Table_nP4hJPX1wh2W3fpo" } ] } ], "sourceDataSourceSettings": [ { "dataSourceProperties": { "timeZone": "Asia/Shanghai" }, "dataSourceName": "autotest_hologres" } ], "jobSettings": { "runtimeSettings": [ ], "ddlHandlingSettings": [ ], "columnDataTypeSettings": [ ], "cycleScheduleSettings": { }, "channelSettings": { "destinationChannelSettings": { "conflictMode": "replace", "dynamicColumnAction": "replay", "writeMode": "replay" }, "sourceChannelSettings": { } } }, "destinationDataSourceType": "Hologres", "transformationRules": [ { "ruleTargetType": "Table", "ruleName": "SourceSchema_Table_BStf8aXPSCJjOWGe", "ruleActionType": "SourceSchema", "ruleExpression": { "columns": [ { "name": "id", "category": "normal", "type": "BIGINT" }, { "name": "decimal", "category": "normal", "type": "DECIMAL" } ] } }, { "ruleTargetType": "Schema", "ruleName": "Rename_Schema_3qWNOIsljtInvKJy", "ruleActionType": "Rename", "ruleExpression": { "expression": "public" } }, { "ruleTargetType": "Table", "ruleName": "Rename_Table_o3PVQq1aIKDGoVVW", "ruleActionType": "Rename", "ruleExpression": { "expression": "auto_holo_2661647_dst" } }, { "ruleTargetType": "Table", "ruleName": "DefineDstTableSettings_Table_BhJltOmOCIc81fzi", "ruleActionType": "DefineDstTableSettings", "ruleExpression": { "ddlString": "BEGIN; CREATE TABLE IF NOT EXISTS public.auto_holo_2661647_dst ( id BIGINT PRIMARY KEY, "decimal" DECIMAL(38,18) ); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'time_to_live_in_seconds', '3153600000'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'orientation', 'column'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'binlog.level', 'replica'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'binlog.ttl', '2592000'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'bitmap_columns', '"text","char","varchar"'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'dictionary_encoding_columns', '"text":auto,"bytea":auto,"char":auto,"varchar":auto'); CALL SET_TABLE_PROPERTY('public.auto_holo_2661647_dst', 'distribution_key', '"id"'); COMMIT; ", "ddlType": "STRUCT" } }, { "ruleTargetType": "Table", "ruleName": "ColumnMapping_Table_nP4hJPX1wh2W3fpo", "ruleActionType": "ColumnMapping", "ruleExpression": { "columnMapping": [ { "sourceColName": "id", "dstColName": "id" }, { "sourceColName": "decimal", "dstColName": "decimal" } ] } } ], "migrationType": "FullAndRealtimeIncremental", "destinationDataSourceSettings": [ { "dataSourceProperties": { }, "dataSourceName": "autotest_hologres" } ], "sourceDataSourceType": "Hologres" }

Response elements

Element	Type	Description	Example
	object	Schema of Response
Id	integer	The ID of the data integration task.	11792
RequestId	string	The request ID. You can use this ID to locate logs and troubleshoot issues.	4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB
DIJobId `deprecated`	integer	[Deprecated] Use the Id response parameter instead.	11792

Examples

Success response

JSON format

{
  "Id": 11792,
  "RequestId": "4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB",
  "DIJobId": 11792
}

Error codes

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.