This topic describes the data types and parameters that LogHub Writer supports and how to configure it by using the code editor.

LogHub Writer allows you to transfer data from a Data Integration reader to LogHub by using Log Service SDK for Java.
Note LogHub does not ensure idempotence. Rerunning a node after the node fails may result in redundant data.

LogHub Writer obtains data from a Data Integration reader and converts the data types that Data Integration supports to STRING. When the number of the data records reaches the value specified for the batchSize parameter, LogHub Writer sends the data records to LogHub at a time by using Log Service SDK for Java. The default value of the batchSize parameter is 1,024. The maximum value is 4,096.

Data types

The following table describes the data types that LogHub Writer supports.
Data Integration data type LogHub data type
LONG STRING
DOUBLE STRING
STRING STRING
DATE STRING
BOOLEAN STRING
BYTES STRING

Parameters

Parameter Description Required Default value
endpoint The endpoint of Log Service. Yes N/A
accessKeyId The AccessKey ID of the account that you can use to connect to Log Service. Yes N/A
accessKeySecret The AccessKey secret of the account that you can use to connect to Log Service. Yes N/A
project The name of the destination Log Service project. Yes N/A
logstore The name of the destination Logstore. Yes N/A
topic The name of the destination topic. No An empty string.
batchSize The number of data records to write to LogHub at a time. Default value: 1,024.
Note The size of the data to write to LogHub at a time cannot exceed 5 MB. Change the value of this parameter based on the size of a single data record.
No 1,024

By default, LogHub Writer writes 1,024 records to LogHub at a time. You can change the value as required.

column The columns in each data record. Yes N/A

Configure LogHub Writer by using the codeless UI

The codeless user interface (UI) is not supported for LogHub Writer.

Configure LogHub Writer by using the code editor

You can use the code editor to configure LogHub Writer. For more information, see Create a sync node by using the code editor.

The following example shows how to configure a sync node to write data to LogHub. For more information about the parameters, see the preceding parameter description.
{
    "type": "job",
    "version": "2.0", // The version number.
    "steps": [
        { 
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "loghub",// The writer type.
            "parameter": {
                "datasource": "",// The connection name.
                "column": [// The columns in each data record.
                    "col0",
                    "col1",
                    "col2",
                    "col3",
                    "col4",
                    "col5"
                ],
                "topic": "",// The name of the destination topic.
                "batchSize": "1024",// The number of data records to write at a time.
                "logstore": ""// The name of the destination Logstore.
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": "" // The maximum number of dirty data records allowed.
        },
        "speed": {
            "concurrent": 3, // The maximum number of concurrent threads.
            "throttle": false// Specifies whether to enable bandwidth throttling. A value of false indicates that the bandwidth is not throttled. A value of true indicates that the bandwidth is throttled. The maximum transmission rate takes effect only if you set this parameter to true.
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}