DataWorks:Write data to a MongoDB data source via batch synchronization

Prepare the MaxCompute source table

1. Create a partitioned table named test_write_mongo with the partition field pt.

   CREATE TABLE IF NOT EXISTS test_write_mongo(
       id STRING ,
       col_string STRING,
       col_int int,
       col_bigint bigint,
       col_decimal decimal,
       col_date DATETIME,
       col_boolean boolean,
       col_array string
   ) PARTITIONED BY (pt STRING) LIFECYCLE 10;

2. Add a partition with the value 20230215.

   insert into test_write_mongo partition (pt='20230215')
   values ('11','name11',1,111,1.22,cast('2023-02-15 15:01:01' as datetime),true,'1,2,3');

3. Verify the table.

   SELECT * FROM test_write_mongo
   WHERE pt='20230215';

Prepare the MongoDB destination collection

This example uses ApsaraDB for MongoDB. Create a collection named test_write_mongo.

db.createCollection('test_write_mongo')

Step 1: Add a MongoDB data source

Add a MongoDB data source and verify network connectivity between the data source and the exclusive resource group for Data Integration. For details, see Configure a MongoDB data source.

Step 2: Create and configure an offline sync node

In DataStudio, create an offline sync node and configure the source and destination. For parameters not covered below, use the default values. For a full parameter reference, see Configure a sync task in the codeless UI.

Configure network connectivity

Select the MongoDB and MaxCompute data sources you created. Select the exclusive resource group for Data Integration you configured. Then test the network connectivity.

Select data sources

Select the MaxCompute partitioned table as the source and the MongoDB collection as the destination.

The following parameters control write behavior:

If Write mode is set to Yes and you specify a field other than _id as the business primary key, a runtime error may occur: After applying the update, the (immutable) field '_id' was found to have been altered to _id: "2". This happens when the data contains records whose _id does not match the replaceKey. For details, see Error: After applying the update, the (immutable) field '_id' was found to have been altered to _id: "2".

{"name":"id","type":"string"}
{"name":"col_string","type":"string"}
{"name":"col_int","type":"long"}
{"name":"col_bigint","type":"long"}
{"name":"col_decimal","type":"double"}
{"name":"col_date","type":"date"}
{"name":"col_boolean","type":"bool"}
{"name":"col_array","type":"array","splitter":","}

Step 3: Submit and publish the sync node

If you are using a standard mode workspace and want to schedule this task periodically, submit and publish the sync node to the production environment. For details, see Publish a task.

Step 4: Run the node and verify the results

Run the sync node. After it completes successfully, verify the synchronized data in the MongoDB collection.

Supported types

The MongoDB Writer plugin supports the following types: INT, LONG, DOUBLE, STRING, BOOL, DATE, and ARRAY.

ARRAY type

Set type to array and configure the splitter property to split a delimited string into a MongoDB array.