All Products
Search

This Product

    E-MapReduce:Flume overview

    Document Center

    E-MapReduce:Flume overview

    Last Updated:Aug 21, 2026

    Apache Flume is a distributed, reliable, and high-availability (HA) system that effectively collects, aggregates, and moves large amounts of log data from various data sources to a central data store.

    Scenarios

    Flume is most commonly used for log collection. It can also transfer other types of data through a custom Source.

    Flume sends data to real-time computing platforms, such as Flink, Spark Streaming, and Storm, and to offline computing platforms, such as MapReduce (MR), Hive, and Presto. Flume can also send data directly to storage systems, such as Hadoop Distributed File System (HDFS), OSS, Kafka, and Elasticsearch. This prepares the data for subsequent analysis and traffic scrubbing.flume2

    Architecture

    A Flume Agent is an instance of Flume. It is a Java Virtual Machine (JVM) process that controls the data stream of Events from producers to consumers. A Flume Agent consists of a Source, a Channel, and a Sink. A Source can write to multiple Channels, and multiple Sinks can read from a single Channel.flume

    Terms

    Name

    Description

    Event

    An Event is the basic unit of data that flows through a Flume Agent. An Event consists of an optional Header map and a byte array that contains the data.

    An example follows.

    --------------------------------
| Header (Map) | Body (byte[]) |
--------------------------------
               Flume Event

    Source

    A Source is a data collector. It collects data from an external data source and sends it in batches to one or more Channels.

    Common Sources include the following:

    • Avro Source: Listens on an Avro port to receive events sent from an Avro client. Avro is a protocol provided by Hadoop for data serialization.

    • Exec Source: Gets data by listening to the output of a command line, such as tail -f /var/log/messages.

    • NetCat TCP Source: Listens on a specified TCP port to get data. It is similar to the Netcat UDP Source.

    • Taildir Source: Monitors multiple files in a directory, records offsets, and prevents data loss. This source is commonly used.

    Channel

    It serves as a buffer queue between the Source and the Sink.

    Common Channels include the following:

    • Memory Channel: Caches data in memory. It provides high performance and is commonly used.

    • File Channel: Caches data in files. It records Checkpoint and DATA files, which provides high reliability but lower performance.

    • JDBC Channel: Caches data in a relational database.

    • Kafka Channel: Caches data using Kafka.

    Sink

    A Sink gets an Event from a Channel and commits it to external storage in a transaction. After the transaction is successfully committed, the Event is removed from the Channel.

    Common Sinks include the following:

    • Logger Sink: Used for testing.

    • Avro Sink: Converts data into Avro Events. It is mainly used to connect multiple Flume Agents.

    • HDFS Sink: Writes data to HDFS. This sink is commonly used.

    • Hive Sink: Writes data to Hive tables or partitions. It uses Hive transactions to write Events.

    • Kafka Sink: Writes data to Kafka.