Message Queue for Apache Kafka can be connected to Filebeat as an input. This topic describes how to use Filebeat
to consume messages in Message Queue for Apache Kafka in a virtual private cloud (VPC) environment.
Background information
Ensure that the following actions are completed:
- A Message Queue for Apache Kafka instance is purchased and deployed. For more information, see Access from a VPC.
- Filebeat is downloaded and installed. For more information, see Download Filebeat.
- Java Development Kit (JDK) 8 is downloaded and installed. For more information, see
Download JDK 8.
Step 1: Obtain an endpoint
Filebeat establishes a connection to Message Queue for Apache Kafka by using a Message Queue for Apache Kafka endpoint.
- Log on to the Message Queue for Apache Kafka console.
- In the left-side navigation pane, click Instances.
- On the Instance Details page, select the instance to be connected to Filebeat as an input.
- In the Basic Information section, obtain the endpoint of the instance.
Step 2: Create a topic
Perform the following operations to create a topic for storing messages.
- In the left-side navigation pane of the Message Queue for Apache Kafka console, click Topics.
- On the Topics page, click Create Topic.
- In the Create Topic dialog box, enter the topic information and click Create.
Step 3: Send messages
Perform the following operations to send messages to the topic you created.
- On the Topics page of the Message Queue for Apache Kafka console, find the topic you created and click Send Message in the Actions column.
- In the Send Message dialog box, enter the message information and click Send.
Step 4: Create a consumer group
Perform the following operations to create a consumer group for Filebeat.
- In the left-side navigation pane of the Message Queue for Apache Kafka console, click Consumer Groups.
- On the Consumer Groups page, click Create Consumer Group.
- In the Create Consumer Group dialog box, enter the consumer group information and click Create.
Step 5: Use Filebeat to consume messages
Start Filebeat on the server where Filebeat has been installed to consume messages
from the created topic.
- Run the cd command to switch to the installation directory of Filebeat:
- Create the input.yml configuration file.
- Run the
vim input.conf
command to create an empty configuration file.
- Press the i key to go to the insert mode.
- Enter the following content:
filebeat.inputs:
- type: kafka
hosts:
- 192.168.XX.XX:9092
- 192.168.XX.XX:9092
- 192.168.XX.XX:9092
topics: ["filebeat_test"]
group_id: "filebeat_group"
output.console:
pretty: true
Parameter |
Description |
Example |
type |
The input type of Filebeat. |
kafka |
hosts |
Message Queue for Apache Kafka supports the following VPC endpoints:
- Default endpoint
- Simple Authentication and Security Layer (SASL) endpoint
|
- 192.168.XX.XX:9092
- 192.168.XX.XX:9092
- 192.168.XX.XX:9092 |
topics |
The name of the topic. |
filebeat_test |
group_id |
The name of the consumer group. |
filebeat_group |
For more information about parameter settings, see Kafka input plugin.
- Press the Esc key to return to the command line mode.
- Press the : key to enter the bottom line mode. Type wq, and then press Enter to save the file and exit.
- Run the following command to consume messages:
./filebeat -c ./input.yml