ApsaraMQ for Kafka:Create an Elasticsearch sink connector

Step 1: Configure basic information

In the Configure Basic Information step, set the connector name and review the instance details.

Click Next.

Step 2: Configure the source service

In the Configure Source Service step, select Message Queue for Apache Kafka as the source service and configure the following parameters.

Click Configure Runtime Environment to expand additional parameters.

Step 3: Configure the destination service

In the Configure Destination Service step, select Elasticsearch as the destination service and configure the following parameters.

Click Create.

Deploy the connector

After creation, the connector appears on the Connectors page. Click Deploy in the Actions column to start the connector.

Send a test message

1. On the Connectors page, find the connector and click Test in the Actions column.

2. In the Send Message panel, set Method of Sending to Console.

3. In the Message Key field, enter a key, for example, demo.

4. In the Message Content field, enter a JSON body, for example:

      {"key": "test"}

5. For Send to Specified Partition, click Yes and enter a Partition ID (for example, 0) to target a specific partition, or click No to let the system assign one. To look up partition IDs, see View partition status.

You can also send test messages through Docker or an SDK. Select the corresponding option in the Method of Sending field and follow the on-screen instructions.

Check the Elasticsearch index

1. Log on to the Kibana console.

2. Run the following query to search the target index:

      GET /<index_name>/_search

3. Confirm that the response contains the message you sent. A successful response looks like this:

      {
        "took": 8,
        "timed_out": false,
        "_shards": {
          "total": 5,
          "successful": 5,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": {
            "value": 1,
            "relation": "eq"
          },
          "max_score": 1.0,
          "hits": [
            {
              "_index": "product_****",
              "_type": "_doc",
              "_id": "TX3TZHgBfHNEDGoZ****",
              "_score": 1.0,
              "_source": {
                "msg_body": {
                  "key": "test",
                  "offset": 2,
                  "overflowFlag": false,
                  "partition": 2,
                  "timestamp": 1616599282417,
                  "topic": "dv****",
                  "value": "test1",
                  "valueSize": 8
                },
                "doc_as_upsert": true
              }
            }
          ]
        }
      }