This topic describes how to connect Message Queue for Apache Kafka to Filebeat.

Filebeat

Filebeat is a lightweight shipper for forwarding and centralizing log data. Filebeat can monitor a specified log file or location, collect log events, and forward them to Elasticsearch or Logstash for indexing. Filebeat works as follows:

  1. Filebeat starts one or more inputs to search for log data in the specified location.
  2. Filebeat starts a harvester for each log it found. Each harvester reads a single log and sends the log data to libbeat.
  3. libbeat aggregates the data and then sends the aggregated data to the configured output.

Connection advantages

Connecting Message Queue for Apache Kafka to Filebeat brings the following advantages:

  • Asynchronous processing: prevents burst traffic.
  • Application decoupling: ensures that a downstream exception does not affect the upstream.
  • Reduce overheads: reduces the resource overhead of Filebeat.

Connection plan

Message Queue for Apache Kafka can be connected to Filebeat in the following ways: