This topic describes the data types and parameters that are supported by LogHub (SLS) Writer and how to configure LogHub (SLS) Writer by using the codeless user interface (UI) and code editor.

LogHub (SLS) Writer can transfer data from a reader to LogHub (SLS) by using Log Service SDK for Java.
Note LogHub (SLS) does not ensure idempotence. If you rerun a failed node, redundant data may be generated.

LogHub (SLS) Writer obtains data from a reader and converts the data types supported by Data Integration to STRING. When the number of the data records reaches the value specified for the batchSize parameter, LogHub (SLS) Writer sends the data records to LogHub at a time by using Log Service SDK for Java. The default value of the batchSize parameter is 1024. The maximum value is 4096.

Data types

The following table lists the data types that are supported by LogHub (SLS) Writer.
Data Integration data type LogHub (SLS) data type
LONG STRING
DOUBLE STRING
STRING STRING
DATE STRING
BOOLEAN STRING
BYTES STRING

Parameters

Parameter Description Required Default value
endpoint The endpoint of the Log Service project. The endpoint is a URL that you can use to access the project and the log data in the project. The endpoint varies based on the project name and the Alibaba Cloud region where the project resides. For more information about the Log Service endpoints in each region, see Endpoints. Yes No default value
accessKeyId The AccessKey ID of the Alibaba Cloud account that is used to access the Log Service project. Yes No default value
accessKeySecret The AccessKey secret of the Alibaba Cloud account that is used to access the Log Service project. Yes No default value
project The name of the Log Service project to which you want to write data. Yes No default value
logstore The name of the Logstore to which you want to write data. A Logstore is a basic unit that you can use to collect, store, and query log data in Log Service. Yes No default value
topic The name of the topic to which you want to write data. No Empty string
batchSize The number of data records to write to LogHub (SLS) at a time. Default value: 1024.
Note The size of the data to write to LogHub (SLS) at a time cannot exceed 5 MB. You can change the value of this parameter based on the size of a single data record.
No 1024

(1,024 data records are written to LogHub at a time.)

column The names of columns in each data record. Yes No default value

Configure LogHub (SLS) Writer by using the codeless UI

  1. Configure data sources.
    Configure Source and Target for the synchronization node. Configure data sources
    Parameter Description
    Connection The name of the data source to which you want to write data.
    Logstore The name of the Logstore to which you want to write data. This parameter is equivalent to the logstore parameter that is described in the preceding section.
    Topic The name of the topic to which you want to write data. This parameter is equivalent to the topic parameter that is described in the preceding section.
    Number of batches The number of data records to write to LogHub (SLS) at a time. This parameter is equivalent to the batchSize parameter that is described in the preceding section.
  2. Configure field mappings. This operation is equivalent to setting the column parameter that is described in the preceding section. Fields in the source on the left have a one-to-one mapping with fields in the destination on the right. Field mappings
    Operation Description
    Map Fields with the Same Name Click Map Fields with the Same Name to establish mappings between fields with the same name. The data types of the fields must match.
    Map Fields in the Same Line Click Map Fields in the Same Line to establish mappings between fields in the same row. The data types of the fields must match.
    Delete All Mappings Click Delete All Mappings to remove the mappings that are established.
    Auto Layout Click Auto Layout. Then, the system automatically sorts the fields based on specific rules.

Configure LogHub (SLS) Writer by using the code editor

For more information about how to configure a synchronization node by using the code editor, see Create a synchronization node by using the code editor.

In the following code, a synchronization node is configured to write data to LogHub (SLS). For more information about the parameters, see the preceding parameter description. 
{
    "type": "job",
    "version": "2.0",// The version number. 
    "steps": [
        { 
            "stepType": "stream",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
            "stepType": "loghub",// The writer type. 
            "parameter": {
                "datasource": "",// The name of the data source to which you want to write data. 
                "column": [// The names of the columns to which you want to write data. 
                    "col0",
                    "col1",
                    "col2",
                    "col3",
                    "col4",
                    "col5"
                ],
                "topic": "",// The name of the topic to which you want to write data. 
                "batchSize": "1024",// The number of data records to write at a time. 
                "logstore": ""// The name of the Logstore to which you want to write data. 
            },
            "name": "Writer",
            "category": "writer"
        }
    ],
    "setting": {
        "errorLimit": {
            "record": ""// The maximum number of dirty data records allowed. 
        },
        "speed": {
            "throttle":true,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
            "concurrent":3, // The maximum number of parallel threads. 
            "mbps":"12"// The maximum transmission rate.
        }
    },
    "order": {
        "hops": [
            {
                "from": "Reader",
                "to": "Writer"
            }
        ]
    }
}