All Products
Search

This Product

    Simple Log Service:Flume consumption

    Document Center

    Simple Log Service:Flume consumption

    Last Updated:Jun 15, 2026

    Integrate Log Service with Apache Flume by using the aliyun-log-flume plugin to write and consume log data.

    Background information

    aliyun-log-flume is a plugin that connects Log Service with Apache Flume, enabling data exchange between Log Service and other systems such as HDFS and Kafka. The plugin provides a sink and a source.

    • Sink: Flume reads data from other data sources and writes it to Log Service.

    • Source: Flume consumes log data from Log Service and writes it to other systems.

    For more information, see aliyun-log-flume.

    Procedure

    1. Download and install Apache Flume.

      For more information, see Apache Flume.

    2. Download the aliyun-log-flume plugin and place it in the /flume/lib directory.

      For more information, see aliyun-log-flume-1.3.jar.

    3. In the /flume/conf directory, create a configuration file named flumejob.conf.

      • For sink configurations and examples, see Sink.

      • For source configurations and examples, see Source.

    4. Start Flume.

    Sink

    A sink writes data from other data sources to Log Service and supports two parsing formats:

    • SIMPLE: writes the entire Flume event as a single field to Log Service.

    • DELIMITED: treats the Flume event as separator-delimited data, parses it into fields based on the configured column names, and then writes them to Log Service.

    The following table describes the sink parameters.

    Parameter

    Required

    Description

    type

    Yes

    The type of the sink. Set this parameter to com.aliyun.Loghub.flume.sink.LoghubSink.

    endpoint

    Yes

    The endpoint of the Log Service project. Example: http://cn-qingdao.log.aliyuncs.com. Replace the value with the actual endpoint. For more information, see Endpoints.

    project

    Yes

    The name of the Log Service project.

    LogStore

    Yes

    The name of the Logstore.

    accessKeyId

    Yes

    The AccessKey ID of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user. For more information, see Create an AccessKey.

    accessKey

    Yes

    The AccessKey secret of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user. For more information, see Create an AccessKey.

    batchSize

    No

    The number of data entries to write to Log Service in each batch. Default value: 1000.

    maxBufferSize

    No

    The size of the buffer queue. Default value: 1,000 entries.

    serializer

    No

    The event serialization format. Valid values:

    • DELIMITED: sets the serialization format to Delimited mode.

    • SIMPLE: sets the serialization format to Simple mode. This is the default mode.

    • JSON: sets the serialization format to JSON mode.

    • Custom serializer: The fully qualified name of a custom serialization class.

    columns

    No

    This parameter is required when serializer is set to DELIMITED. It specifies a list of field names separated by commas (,). The order of the fields must match the order in your data.

    separatorChar

    No

    This parameter is used when serializer is set to DELIMITED. It specifies the separator for the data. The separator must be a single character. Default value: comma (,).

    quoteChar

    No

    This parameter is used when serializer is set to DELIMITED. It specifies the quote character. Default value: double quotation mark (").

    escapeChar

    No

    This parameter is used when serializer is set to DELIMITED. It specifies the escape character. Default value: double quotation mark (").

    useRecordTime

    No

    Specifies whether to use the data's timestamp field as the log time. The default value is false, which uses the current time.

    For sink configuration examples, see GitHub.

    Source

    A source consumes log data from Log Service and sends it to other systems. Two output formats are supported:

    • DELIMITED: writes data to Flume in delimited format.

    • JSON: writes data to Flume in JSON format.

    The following table describes the source parameters.

    Parameter

    Required

    Description

    type

    Yes

    The type of the source. Set this parameter to com.aliyun.Loghub.flume.source.LoghubSource.

    endpoint

    Yes

    The endpoint of the Log Service project. Example: http://cn-qingdao.log.aliyuncs.com. Replace the value with the actual endpoint. For more information, see Endpoints.

    project

    Yes

    The name of the Log Service project.

    LogStore

    Yes

    The name of the Logstore.

    accessKeyId

    Yes

    The AccessKey ID of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user. For more information, see Create an AccessKey.

    accessKey

    Yes

    The AccessKey secret of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user. For more information, see Create an AccessKey.

    heartbeatIntervalMs

    No

    The heartbeat interval between the client and Log Service. Default value: 30,000 ms.

    fetchIntervalMs

    No

    The data fetch interval. Default value: 100 ms.

    fetchInOrder

    No

    Specifies whether to consume data in sequential order. Default value: false.

    batchSize

    No

    The number of data entries to read in each batch. Default value: 100.

    consumerGroup

    No

    The name of the consumer group used for reading data.

    initialPosition

    No

    The initial position from which to start reading data. Valid values: begin, end, and timestamp. Default value: begin.

    Note

    If a server-side checkpoint exists, it takes precedence.

    timestamp

    No

    This parameter is required when initialPosition is set to timestamp. Specify the time in the UNIX timestamp format.

    deserializer

    Yes

    The event deserialization format. Valid values:

    • DELIMITED: sets the deserialization format to Delimited mode. This is the default mode.

    • JSON: sets the deserialization format to JSON mode.

    • Custom deserializer: The fully qualified name of a custom deserialization class.

    columns

    No

    This parameter is required when deserializer is set to DELIMITED. It specifies a list of field names separated by commas (,). The order of the fields must match the order in your data.

    separatorChar

    No

    This parameter is used when deserializer is set to DELIMITED. It specifies the separator for the data. The separator must be a single character. Default value: comma (,).

    quoteChar

    No

    This parameter is used when deserializer is set to DELIMITED. It specifies the quote character. Default value: double quotation mark (").

    escapeChar

    No

    This parameter is used when deserializer is set to DELIMITED. It specifies the escape character. Default value: double quotation mark (").

    appendTimestamp

    No

    This parameter is used when deserializer is set to DELIMITED. It specifies whether to automatically append the timestamp as a field to the end of each row. Default value: false.

    sourceAsField

    No

    This parameter is used when deserializer is set to JSON. It specifies whether to add the log source as a field named __source__. Default value: false.

    tagAsField

    No

    This parameter is used when deserializer is set to JSON. It specifies whether to add log tags as fields. Field names are formatted as __tag__:{tag_name}. Default value: false.

    timeAsField

    No

    This parameter is used when deserializer is set to JSON. It specifies whether to add the log time as a field named __time__. Default value: false.

    useRecordTime

    No

    Specifies whether to use the original log time from Log Service for the Flume event's timestamp. If false, the current system time is used. The default value is false.

    For source configuration examples, see GitHub.