This topic describes the data types and parameters supported by LogHub Writer and how to configure it by using the code editor.

LogHub Writer allows you to transfer data from a Data Integration reader to LogHub through Log Service Java SDK.
Note LogHub does not guarantee idempotence. Rerunning a node after the node fails may result in redundant data.

LogHub Writer obtains data from a Data Integration reader and converts the data types supported by Data Integration to String. When the number of the data records reaches the value specified for the batchSize parameter, LogHub Writer sends the data records to LogHub at a time through Log Service Java SDK. LogHub Writer sends 1,024 data records at a time by default. The batchSize parameter can be set to 4096 at most.

Data types

The following table lists the data types supported by LogHub Writer.
Data Integration data type LogHub data type
LONG STRING
DOUBLE STRING
STRING STRING
DATE STRING
BOOLEAN STRING
BYTES STRING

Parameters

Parameter Description Required Default value
endpoint The endpoint for accessing Log Service. Yes None
accessKeyId The AccessKey ID for accessing Log Service. Yes None
accessKeySecret The AccessKey secret for accessing Log Service. Yes None
project The name of the destination Log Service project. Yes None
logstore The name of the destination Logstore. Yes None
topic The name of the destination topic. No Empty string
batchSize The number of data records to write to LogHub at a time. Default value: 1024.
Note The size of the data to write to LogHub at a time cannot exceed 5 MB. Change the value of this parameter based on the size of a single data record.
No 1024

By default, LogHub Writer writes 1,024 records to LogHub at a time. You can change the value as required.

column The columns in each data record. Yes None

Configure LogHub Writer by using the codeless UI

Currently, the codeless user interface (UI) is not supported for LogHub Writer.

Configure LogHub Writer by using the code editor

In the following code, a node is configured to write data to LogHub. For more information about the parameters, see the preceding parameter description.
{
    "type": "job",
    "version": "2.0",// The version number.
    "steps": [
        { 
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "loghub",// The writer type.
            "parameter": {
                "datasource": "",// The connection name.
                "column": [// The columns in each data record.
                    "col0",
                    "col1",
                    "col2",
                    "col3",
                    "col4",
                    "col5"
                ],
                "topic": "",// The name of the destination topic.
                "batchSize": "1024",// The number of data records to write at a time.
                "logstore": ""// The name of the destination Logstore.
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The maximum number of dirty data records allowed.
        },
        "speed": {
            "concurrent": 3,// The maximum number of concurrent threads.
            "throttle": false,// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}