A Message Queue for Apache Kafka instance can be connected to Logstash as an input. This topic describes how to use Logstash to consume messages from Message Queue for Apache Kafka over the Internet.
Prerequisites
- A Message Queue for Apache Kafka instance is purchased and deployed. For more information, see Purchase and deploy an instance that allows access from the Internet and a VPC.
- Logstash is downloaded and installed. For more information, see Download Logstash.
- Java Development Kit (JDK) 8 is downloaded and installed. For more information, see the Java 8 tab.
Step 1: Obtain the public endpoint
Logstash establishes a connection to Message Queue for Apache Kafka by using a Message Queue for Apache Kafka endpoint. The username and password of your Message Queue for Apache Kafka instance are required for identity authentication.
Step 2: Create a topic
Perform the following operations to create a topic for storing messages:
Step 3: Send messages
Perform the following operations to send messages to the topic that you created:
- Log on to the Message Queue for Apache Kafka console.
- In the Resource Distribution section of the Overview page, select the region where your instance resides.
- On the Instances page, click the name of the instance that you want to manage.
- In the left-side navigation pane, click Topics.
- On the Topics page, find the topic based on which you want to send and consume messages, and choose in the Actions column.
- In the Start to Send and Consume Message panel, set the parameters or use the method as prompted to send a test message.
- Set the Method of Sending parameter to Console.
- In the Message Key field, enter the key of the test message, such as demo.
- In the Message Content field, enter the content of the test message, such as {"key": "test"}.
- Set the Send to Specified Partition parameter to specify whether to send the test message to a specific partition.
- If you want to send the test message to a specific partition, click Yes and enter the partition ID, such as 0, in the Partition ID field. For more information about how to query partition IDs, see View partition status.
- If you do not want to send the test message to a specific partition, click No.
- Use Message Queue for Apache Kafka SDKs or run docker commands to consume the test message as prompted.
- Set the Method of Sending parameter to Docker and run a Docker container.
- Run the docker commands provided in the Run the Docker container to produce a sample message section to send the test message.
- Run the docker commands provided in the How do I consume a message after the message is sent? section to consume the test message.
- Set the Method of Sending parameter to SDK, select a programming language or a framework, and then select an access method to use the corresponding SDK to send and consume messages.
- Set the Method of Sending parameter to Console.
Step 4: Create a consumer group
Perform the following operations to create a consumer group for Logstash.
Step 5: Use Logstash to consume messages
Start Logstash on the server where Logstash is installed, and consume messages from the created topic.
References
For more information about parameter settings, see Kafka input plugin.