This topic describes how to connect Message Queue for Apache Kafka to Filebeat.


Filebeat is a lightweight shipper for forwarding and centralizing log data. Filebeat can monitor a specified log file or location, collect log events, and forward them to Elasticsearch or Logstash for indexing. Filebeat works in the following way:

  1. Filebeat starts one or more inputs to search for log data in the specified location.
  2. Filebeat starts a harvester for each log it found. Each harvester reads a single log and sends the log data to libbeat.
  3. libbeat aggregates the data and then sends the aggregated data to the configured output.

Connection advantages

Connecting Message Queue for Apache Kafka to Filebeat brings the following advantages:

  • Asynchronous processing: prevents burst traffic.
  • Application decoupling: ensures that a downstream exception does not affect the upstream.
  • Overhead reduction: reduces the resource overhead of Filebeat.

Connection methods

Message Queue for Apache Kafka can be connected to Filebeat in the following ways: