Message Queue for Apache Kafka can be connected to Filebeat as an output. This topic describes how to use Filebeat to send messages to Message Queue for Apache Kafka in a virtual private cloud (VPC) environment.

Prerequisites

Ensure that you have completed the following operations:
  • You have purchased and deployed a Message Queue for Apache Kafka instance. For more information, see Access from VPC.
  • You have downloaded and installed Filebeat. For more information, see Download Filebeat.
  • You have downloaded and installed Java Development Kit (JDK) 8. For more information, see Download JDK 8.

Step 1: Obtain an endpoint

Filebeat establishes a connection to Message Queue for Apache Kafka through a Message Queue for Apache Kafka endpoint.

  1. Log on to the Message Queue for Apache Kafka console.
  2. In the left-side navigation pane, click Instances.
  3. On the Instance Details page, select the instance to be connected to Filebeat as an output.
  4. In the Basic Information section, obtain the endpoint of the instance.
    Note For information about the differences between endpoints, see Endpoint comparison.

Step 2: Create a topic

Perform the following operations to create a topic for storing messages.

  1. In the left-side navigation pane of the Message Queue for Apache Kafka console, click Topics.
  2. On the Topics page, click Create Topic.
  3. In the Create Topic dialog box, enter the topic information and click Create.

Step 3: Use Filebeat to send a message

Start Filebeat on the server where Filebeat has been installed to send a message to the topic you created.

  1. Run the following command to switch to the installation directory of Filebeat:
    cd /root/filebeat/filebeat-7.7.0-linux-x86_64
  2. Create the output.conf configuration file.
    1. Run the vim output.conf command.
    2. Press i.
    3. Enter the following content:
      filebeat.inputs:
      - type: stdin
      
      output.kafka:
        hosts: ["192.XX.XX.XX:9092", "192.XX.XX.XX:9092", "192.XX.XX.XX:9092"]
      
        topic: 'filebeat_test'
      
        required_acks: 1
        compression: none
        max_message_bytes: 1000000
      Parameter Description Example
      hosts Message Queue for Apache Kafka supports the following VPC endpoints:
      • Default endpoint
      • SASL endpoint
      192.168.XXX.XXX:9092,192.168.XXX.XXX:9092,192.168.XXX.XXX:9092
      topic The name of the topic. filebeat_test
      required_acks The reliability level of ACK. Valid values:
      • 0: no response
      • 1: waits for local commit
      • -1: waits for all replicas to commit
      Default value: 1.
      1
      compression The data compression codec. Valid values:
      • none
      • snappy
      • lz4
      • gzip
      Default value: gzip.
      none
      max_message_bytes The maximum size of a message. Unit: byte. Default value: 1000000. The value must be less than the maximum message size you have configured for Message Queue for Apache Kafka. 1000000

      For more information about parameter settings, see Kafka output plugin.

    4. Press Esc, enter :wq, and then press Enter.
  3. Send a message to the topic you created.
    1. Run the ./filebeat -c ./output.yml command.
    2. Enter test and then press Enter.

Step 4: View the partitions of the topic

Perform the following operations to view the message that was sent to the topic.

  1. In the left-side navigation pane of the Message Queue for Apache Kafka console, click Topics.
  2. On the Topics page, select the instance that is to be connected to Filebeat as an output, find the topic to which the message was sent, and click Partition Status in the Actions column.
  3. On the Partition Status page, click Refresh.

Step 5: Query the message by offset

You can query the sent message based on its partition ID and offset information.

  1. In the left-side navigation pane of the Message Queue for Apache Kafka console, click Message Query.
  2. On the Message Query page, click Query by Offset.
  3. Enter the topic of the sent message, select the partition ID of the sent message from the drop-down list, enter the offset of the sent message, and click Search.
  4. Optional:Find the search result, and click Message Details in the Actions column.