Migrate data from ApsaraDB for MongoDB to LindormTable - Lindorm

DataWorks is a platform as a service (PaaS) from Alibaba Cloud that supports a wide range of computing and storage engines. This topic describes how to use DataWorks to migrate offline data from ApsaraDB for MongoDB to LindormTable.

Background information

For more information about DataWorks, see What is DataWorks?.

Usage notes

When you migrate offline data from ApsaraDB for MongoDB to LindormTable, you must flatten the nested JSON data from MongoDB into a horizontal structure. This process does not involve data transformation.

Note

If you need to transform data during the migration, such as applying an MD5 hash to a primary key in your MongoDB data, follow these steps:

Use DataWorks to migrate the data from ApsaraDB for MongoDB to MaxCompute.
Run SQL statements in MaxCompute to process the data.
Use DataWorks to migrate the processed data from MaxCompute to LindormTable.

Prerequisites

Before you migrate offline data from ApsaraDB for MongoDB to LindormTable, complete the following preparatory steps.

Prepare the data in ApsaraDB for MongoDB. The following code shows a sample document:

{
   "id" : ObjectId("624573dd7c0e2eea4cc8****"),
   "title" : "MongoDB Tutorial",
   "description" : "MongoDB is a NoSQL database",
   "by" : "Beginner's Guide",
   "url" : "http://www.runoob.com",
   "map" : {
        "a" : "mapa",
        "b" : "mapb"
    },
   "likes" : 100
}

Prepare the table schema in LindormTable. The following code provides a sample schema:

CREATE TABLE t1(title varchar, desc varchar, by1 varchar, url varchar, a varchar, b varchar, likes int, primary key(title));

Configure a DataX task using the Data Integration service of DataWorks. For more information, see Configure a DataX task by using DataWorks.

Procedure

Configure an ApsaraDB for MongoDB data source in the DataWorks console. For more information, see Configure an ApsaraDB for MongoDB data source.

Configure a batch synchronization task using the code editor. For more information, see Configure a batch synchronization task in the code editor.

Create a workflow.
1. Log on to the DataWorks console.
2. In the left-side navigation pane, click Workspaces.
3. Select the region where your workspace resides. In the Actions column of the target workspace, choose Shortcuts > Data Development.
4. On the Data Development page, hover over the icon and click Create Workflow.
5. In the Create Workflow dialog box, enter a Workflow Name and a Description.
  Note
  The workflow name must be 128 characters or less and can contain only uppercase letters, lowercase letters, digits, underscores (_), and periods (.).
6. Click Create.
Create a batch synchronization node.
1. Expand the workflow and right-click Data Integration.
2. Choose Create Node > Batch Synchronization.
3. In the Create Node dialog box, enter a Name for the node.
  Note
  The node name must be 128 characters or less and can contain only uppercase letters, lowercase letters, digits, underscores (_), and periods (.).
4. Click Submit.

Configure the reader and writer.

After the batch synchronization node is created, click the Script Conversion (</>) icon in the toolbar of the batch synchronization task configuration page.
In the dialog box that appears, click Confirm to go to the code editor.

A basic reader configuration is automatically generated in the code editor. You can manually configure the reader and writer data sources, and specify the tables to be synchronized. The following code provides a sample configuration:

Note

For more information about the parameters for MongoDB Reader, see MongoDB Reader.
For more information about the parameters for Lindorm Writer, see Lindorm Writer.

{
    "type": "job",
    "version": "2.0",
    "steps": [
        {
            "stepType": "mongodb",
            "parameter": {
                "datasource": "test_mongo",   // The name of the ApsaraDB for MongoDB data source.
                "column": [
                    {
                        "name": "title",
                        "type": "string"
                    },
                    {
                        "name": "description",
                        "type": "string"
                    },
                    {
                        "name": "by",
                        "type": "string"
                    },
                    {
                        "name": "url",
                        "type": "string"
                    },
                    {
                        "name": "map.a",
                        "type": "document.string"
                    },
                    {
                        "name": "map.b",
                        "type": "document.string"
                    },
                    {
                        "name": "likes",
                        "type": "int"
                    }
                ],
                "collectionName": "testdatax"
            },
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "lindorm",
            "parameter": {
                "configuration":  {
                    "lindorm.client.seedserver": "ld-xxxx-proxy-lindorm.lindorm.rds.aliyuncs.com:30020",
                    "lindorm.client.username": "root",
                    "lindorm.client.namespace": "test",
                    "lindorm.client.password": "root"
                },
                "nullMode": "skip",
                "datasource": "",
                "writeMode": "api",
                "envType": 1,
                "columns": [
                    "title",
                    "desc",
                    "by",
                    "url",
                    "a",
                    "b",
                    "likes"
                ],
                "dynamicColumn": "false",
                "table": "t1",
                "encoding": "utf8"
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "executeMode": null,
        "errorLimit": {
            "record": ""
        },
        "speed": {
            "concurrent": 2,
            "throttle": false
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}

Save the configuration and click the icon in the upper-left corner of the code editor. You can view the data migration progress on the Runtime Log tab.