This topic describes how to connect Message Queue for Apache Kafka to Filebeat.
Filebeat is a lightweight shipper for forwarding and centralizing log data. Filebeat can monitor a specified log file or location, collect log events, and forward them to Elasticsearch or Logstash for indexing. Filebeat works in the following way:
- Filebeat starts one or more inputs to search for log data in the specified location.
- Filebeat starts a harvester for each log it found. Each harvester reads a single log and sends the log data to libbeat.
- libbeat aggregates the data and then sends the aggregated data to the configured output.
Connecting Message Queue for Apache Kafka to Filebeat brings the following advantages:
- Asynchronous processing: prevents burst traffic.
- Application decoupling: ensures that a downstream exception does not affect the upstream.
- Overhead reduction: reduces the resource overhead of Filebeat.
Message Queue for Apache Kafka can be connected to Filebeat in the following ways: