Kafka Writer allows you to write data to Kafka by using Kafka SDK for Java. This topic describes how Kafka Writer works, parameters supported by Kafka Writer, and how to configure Kafka Writer by using the code editor.

Notice Kafka Writer supports only exclusive resource groups for data integration.

Apache Kafka is a fast, scalable, high-throughput, and distributed messaging system that supports fault tolerance. This system is used to publish and subscribe to messages. Kafka provides built-in partitions and supports data replicas. It can be used to process a large number of messages. Kafka Writer is in public preview.

How it works

Kafka Writer writes data to Kafka by using the following version of Kafka SDK for Java:
<dependency>
   <groupId>org.apache.kafka</groupId>
   <artifactId>kafka-clients</artifactId>
   <version>2.0.0</version>
</dependency>

Parameters

Parameter Description Required
server The address of the Kafka broker, in the format of ip:port. Yes
topic The name of the Kafka topic to which data is written. Kafka maintains feeds of messages in categories called topics.

Each message that is published to the Kafka cluster is assigned to a topic. Each topic contains a group of messages.

Yes
keyIndex The column that is specified as the key. Yes
valueIndex The column that is specified as the value. If this parameter is not specified, all columns are concatenated by using the delimiter that is specified by fieldDelimiter to form the value. No
fieldDelimiter The delimiter that is used to separate columns when data is written to the Kafka topic. Default value: \t. No
keyType The data type of the key in the Kafka topic. Valid values: BYTEARRAY, DOUBLE, FLOAT, INTEGER, LONG, and SHORT. Yes
valueType The data type of the value in the Kafka topic. Valid values: BYTEARRAY, DOUBLE, FLOAT, INTEGER, LONG, and SHORT. Yes
batchSize The number of data records that are written to the Kafka topic at a time. Default value: 1024. No

Configure Kafka Writer by using the codeless UI

This method is not supported.

Configure Kafka Writer by using the code editor

You can configure Kafka Writer by using the code editor. For more information, see Create a sync node by using the code editor.

The following code shows a configuration example:
{
    "type":"job",
    "version":"2.0",// The version number.
     "steps":[
        {
            "stepType":"stream",
            "parameter":{},
            "name":"Reader",
            "category":"reader"
        },
        {
            "stepType":"Kafka",// The writer type.
             "parameter":{
                   "server": "ip:9092",// The address of the Kafka broker.
                    "keyIndex": 0, // The column that is used as the key. 
                   "valueIndex": 1, // The column that is used as the value. You can specify only one column. If you do not specify this parameter, all columns are used as the value.
        // If you want to use the second, third, and fourth columns in a MaxCompute table as the value, cleanse and integrate the data in the table. Then, create a MaxCompute table, write the processed data to the new table, and use the new table for data synchronization.
                    "keyType": "Integer", // The data type of the key in the Kafka topic.
                    "valueType": "Short", // The data type of the value in the Kafka topic.
                    "topic": "t08", // The name of the Kafka topic to which data is written.
                    "batchSize": 1024 // The number of data records to write at a time.
             },
            "name":"Writer",
            "category":"writer"
        }
    ],
    "setting":{
        "errorLimit":{
            "record":"0"// The maximum number of dirty data records allowed.
         },
        "speed":{
            "throttle":false,// Specifies whether to enable bandwidth throttling. The value false indicates that bandwidth throttling is disabled, and the value true indicates that bandwidth throttling is enabled. The concurrent parameter takes effect only when the throttle parameter is set to true.
             "concurrent":1,// The maximum number of concurrent jobs.
         }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
            }
        ]
    }
}