ApsaraMQ for Kafka:Connect Filebeat to ApsaraMQ for Kafka over the internet

Step 1: Get the endpoint and credentials

Filebeat connects through the public SSL endpoint and authenticates with the SASL username and password of your ApsaraMQ for Kafka instance.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where your instance is deployed.

3. On the Instances page, click the name of your instance.

4. On the Instance Details page, collect the following values:

   Value	Location on page	Description
   SSL endpoint	Endpoint Information section	The public endpoint uses port 9093.
   Username	Configuration Information section	The SASL username for authentication.
   Password	Configuration Information section	The SASL password for authentication.

Step 2: Create a topic

Create a topic that Filebeat consumes messages from.

1. In the ApsaraMQ for Kafka console, navigate to your instance and click Topics in the left-side navigation pane.

2. Click Create Topic.

3. In the Create Topic panel, configure the following parameters and click OK.

After the topic is created, it appears on the Topics page.

Step 3: Send test messages

Publish messages to the topic so Filebeat has data to consume.

1. In the ApsaraMQ for Kafka console, go to Topics and click the topic name.

2. In the upper-right corner of the Topic Details page, click Send Message.

3. In the Start to Send and Consume Message panel, choose a sending method:

   • Console: Enter a Message Key (for example, demo) and Message Content (for example, {"key": "test"}). Optionally, set Send to Specified Partition to Yes and enter a Partition ID to target a specific partition. For information about how to query partition IDs, see View partition status. After sending the message, use ApsaraMQ for Kafka SDKs or run the Docker commands displayed in the panel to subscribe to the test message.

   • Docker: Run the Docker commands displayed in the panel.

   • SDK: Select an SDK for your programming language and follow the instructions.

Step 4: Create a consumer group

Create a consumer group for Filebeat to use.

1. In the ApsaraMQ for Kafka console, navigate to your instance and click Groups in the left-side navigation pane.

2. Click Create Group.

3. In the Create Group panel, enter a Group ID and an optional Description, attach tags if needed, and click OK.

The consumer group appears on the Groups page.

Step 5: Download the certificate, configure Filebeat, and start consuming

With the endpoint, topic, and consumer group ready, configure Filebeat to consume messages over SSL.

wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20220826/ytsw/only-4096-ca-cert

filebeat.inputs:
- type: kafka
  hosts:
    - <your-kafka-endpoint-1>:9093
    - <your-kafka-endpoint-2>:9093
    - <your-kafka-endpoint-3>:9093
  username: "<your-instance-username>"
  password: "<your-instance-password>"
  topics: ["<your-topic>"]
  group_id: "<your-group-id>"
  ssl.certificate_authorities: ["<path-to-ca-cert>"]
  ssl.verification_mode: none

output.console:
  pretty: true

Start Filebeat

Run the following command in the Filebeat installation directory:

./filebeat -c ./input.yml

On a successful connection, consumed messages appear in JSON format:

What to do next

• Route to Elasticsearch: Replace the output.console section with an output.elasticsearch configuration to index consumed messages. See Elasticsearch output in the Filebeat documentation.

• Parse message content: Add parsers or processors to your Filebeat configuration to parse JSON or multiline messages before output.