DataWorks:DataHub data source

Batch read/write

The STRING data type supports only UTF-8 encoding. A single STRING column cannot exceed 1 MB.

Real-time read/write

• Real-time synchronization tasks support serverless resource groups.

• When you synchronize data to DataHub in real time, data is sharded based on the hash value of the source data. Records with the same hash value are synchronized to the same shard.

Real-time full-database write

When you run the task, a batch synchronization task first writes the full data to DataHub. Then, a real-time synchronization task starts, synchronizing incremental data from the source to the destination. Data is written based on the following rules:

• Data can be written only to DataHub topics of the TUPLE type. For more information about the TUPLE data type, see Data types.

• When you synchronize data to DataHub in real time, five additional fields are appended to the source table fields. You can also add other fields when you configure the task. For more information about the final message format sent to DataHub, see Appendix: DataHub message formats.

Configure a single-table batch synchronization task

• For the procedure, see Configure a batch synchronization task in wizard mode and Configure a batch synchronization task in script mode.

• For all parameters and script demos for script mode configuration, see Appendix: Script demos and parameter description below.

Configure a single-table real-time synchronization task

Configure a real-time full-database synchronization task

For the procedure, see Configure a real-time full-database synchronization task.

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Use the Code Editor. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demo

{
    "type":"job",
    "version":"2.0",// The version number.
    "steps":[
        {
         "job": {
           "content": [
            {
                "reader": {
                    "name": "DataHubreader",
                    "parameter": {
                        "endpoint": "xxx" // The endpoint of DataHub.
                        "accessId": "xxx", // The AccessKey ID used to access DataHub.
                        "accessKey": "xxx", // The AccessKey secret used to access DataHub.
                        "project": "xxx", // The name of the DataHub project.
                        "topic": "xxx" // The name of the DataHub topic.
                        "batchSize": 1000, // The number of records to read at a time.
                        "beginDateTime": "20180910111214", // The start time for data consumption.
                        "endDateTime": "20180910111614", // The end time for data consumption.
                        "column": [
                            "col0",
                            "col1",
                            "col2",
                            "col3",
                            "col4"
                                  ]
                                }
                           },
                "writer": {
                    "name": "streamwriter",
                    "parameter": {
                        "print": false
                                 }
                            }
             }
           ]
         }
       }
     ],
    "setting":{
        "errorLimit":{
            "record":"0"// The error count.
        },
        "speed":{
            "throttle":true,// Specifies whether to enable throttling. If throttle is set to false, the mbps parameter does not take effect, which means throttling is disabled. If throttle is set to true, throttling is enabled.
            "concurrent":1,// The concurrency.
            "mbps":"12"// The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}

Reader script parameters

Writer script demo

{
    "type": "job",
    "version": "2.0",// The version number.
    "steps": [
        { 
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "datahub",// The plug-in name.
            "parameter": {
                "datasource": "",// The data source.
                "topic": "",// A topic is the smallest unit for subscription and publishing in DataHub. You can use a topic to represent a type or category of streaming data.
                "maxRetryCount": 500,// The maximum number of retries upon task failure.
                "maxCommitSize": 1048576// The data is committed in batches to the destination when the accumulated data buffer reaches the maxCommitSize (in bytes).
                 // DataHub limits a single request to 10,000 records. Exceeding this limit causes a task error. Set this parameter based on the average size per record multiplied by 10,000. For example, if each record is 10 KB, set this parameter to a value lower than 10 × 10,000 KB.
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The error count.
        },
        "speed": {
            "throttle":true,// Specifies whether to enable throttling. If throttle is set to false, the mbps parameter does not take effect, which means throttling is disabled. If throttle is set to true, throttling is enabled.
            "concurrent":20, // The concurrency.
            "mbps":"12"// The throttling rate. 1 mbps = 1 MB/s.
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}

Writer script parameters