ApsaraMQ for Kafka:Message retrieval

Billing

• The message retrieval feature of ApsaraMQ for Kafka is in public preview. ApsaraMQ for Kafka does not charge you for the message retrieval feature during the public preview.

• During the public preview of the message retrieval feature, you are not charged for instances and tables that are automatically created in Tablestore.

Usage notes

• By default, you can enable the message retrieval feature for up to three topics on an ApsaraMQ for Kafka instance at the same time.

• The value in an attribute column of the STRING type in Tablestore cannot exceed 2 MB. A message whose size exceeds 2 MB cannot be synchronized or retrieved.

• Tablestore retains messages that are synchronized from an ApsaraMQ for Kafka instance for the same period of time as ApsaraMQ for Kafka retains messages. After the retention period expires, Tablestore automatically clears the messages and removes the related indexes. For information about the configuration and description of the message retention period in ApsaraMQ for Kafka, see Modify configurations for messages.

  The data expiration policy of Tablestore is different from that of ApsaraMQ for Kafka. Therefore, the messages retrieved by each message retrieval task may be different from the messages that meet the specified search conditions. In this case, the final retrieval result shall prevail.

Procedure for message retrieval

Step 1: Enable the message retrieval feature

After you enable the message retrieval feature for a topic on an ApsaraMQ for Kafka instance, you can retrieve messages in the topic based on your business requirements.

1. Log on to the ApsaraMQ for Kafka console.

2. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

3. In the left-side navigation pane, click Instances.

4. On the Instances page, click the name of the instance that you want to manage.

5. In the left-side navigation pane, click Message Retrieval. On the page that appears, click Enable Message Retrieval.

6. In the Enable Message Retrieval panel, configure the parameters and click OK.

   Note

   The first time you enable the message retrieval feature for a topic on an ApsaraMQ for Kafka instance, a message indicating that you have not enabled the connector feature for the instance appears after you click Enable Message Retrieval. In this case, click OK in the message and then configure the parameters in the Enable Message Retrieval panel.

   

   Parameters for enabling the message retrieval feature

   Parameter	Description	Example
   Data Source Topic	The topic for which you want to enable the message retrieval feature.	test
   Consumer Offset	The offset from which you want to start message consumption. Default value: Latest Offset. Valid values: Earliest Offset: consumes messages from the earliest offset. Latest Offset: consumes messages from the latest offset.	Latest Offset
   Record Time	The time when each retrieved message is recorded. Default value: Instant Time. Valid values: Native Time: the message creation time that is recorded by ApsaraMQ for Kafka. This is the timestamp that is recorded by the producer when the message is sent or the value of the timestamp field that you specify for ProducerRecord. Instant Time: the time when the message is synchronized to Tablestore. This value specifies the time when the message is synchronized to Tablestore instead of when the message is produced. Therefore, the value in the search result may be different from the time when the message is produced.	Native Time

   On the Message Retrieval page, you can view the topic for which you enabled the message retrieval feature.

Step 2: Send a test message

After you enable the message retrieval feature for a topic on an ApsaraMQ for Kafka instance, you can send messages to the topic to test whether the message retrieval task is created.

1. On the Message Retrieval page, find the topic that you want to manage and click Details in the Actions column.

2. In the upper-right corner of the Task Details page, click Test.

3. In the Send Message panel, configure the parameters and click OK.

   1. In the Message Key field, enter the message key. Example: demo.

   2. In the Message Content field, enter the message content. Example: {"key": "test"}.

   3. Configure the Send to Specified Partition parameter to specify whether to send the test message to a specific partition.

      • If you want to send the test message to a specific partition, click Yes and enter the partition ID in the Partition ID field. Example: 0. For information about how to query partition IDs, see View partition status.

      • If you do not want to send the test message to a specific partition, click No.

Step 3: Retrieve messages

1. On the Message Retrieval page, find the topic that you want to manage and click Search in the Actions column.

2. In the Search panel, perform the following operations to specify a search condition: Select an item from the search condition drop-down list, click Add Search Condition, specify the search condition in the Value column, and then click OK.

   

   Note

   • When you specify the search condition, you can select one or more values, including Partition, Offset Range, Time Range, Key, and Value.

   • We recommend that you use the unique business identifier as the key value when you send a message. This way, the message can be exactly matched during the search.

   Search parameters

   Search condition	Description
   Partition	The partition from which you want to retrieve messages.
   Offset Range	The offset range of the messages that you want to retrieve.
   Time Range	The time range within which the messages that you want to retrieve fall. You can select a time range within the last three days or specify a custom time range. The time range that you can specify is accurate to the minute.
   Key	The key or content of the messages that you want to retrieve. The following items describe the rules that you must follow when you specify a search keyword: Phrase matching is used as the search pattern. For example, if the message key is "ApsaraMQ for Kafka is a distributed, high-throughput, and scalable message queue service that is provided by Alibaba Cloud", you can set the search keyword to "distributed" or a combination of "Alibaba Cloud" and "distributed". If the search keyword contains an asterisk (*) or a question mark (?), the wildcard matching pattern is used. The asterisk (*) specifies an arbitrary character string. The question mark (?) specifies an arbitrary character. Letters are not case-sensitive. For example, if the message content is "AliKafkaTest001qaz", you can specify the search keyword as "AliKafkaTe*". To improve retrieval efficiency, we recommend that you do not use keywords that start with a wildcard character. Make sure that strings that contain a wildcard character do not exceed 20 characters in length. For information about the usage and limits of tokenization, phrase matching, and wildcard matching, see Tokenization, Match phrase query, and Wildcard query.
   Value

3. On the Topic Search page, view the retrieved messages.

   

   Note

   • Up to 10 messages can be displayed in the retrieval result. We recommend that you specify accurate search conditions for exact match.

   • Up to 1 KB of content for each retrieved message can be displayed in the ApsaraMQ for Kafka console. If a retrieved message exceeds 1 KB in size, the system automatically truncates the content. If you want to view the complete message content, download the message.

   Parameters in the retrieval result

   Parameter	Description
   Partition	The partition from which the message is retrieved.
   Offset	The message offset.
   Key	The message key. The key is transformed into a string.
   Value	The message value, which is also known as message content. The message value is transformed into a string.
   Created At	The time when the message was created or when the message was synchronized to Tablestore. The message creation time is the timestamp that was recorded by the producer when the message was sent or the value of the timestamp field that you specified for ProducerRecord. Note If you specified a value for the timestamp field, the specified value is displayed. If you did not specify a value for the timestamp field, the local system time when the message was sent is displayed. A value in the format of 1970/x/x x:x:x indicates that the timestamp field is set to 0 or an invalid value.
   Actions	Click Download Key to download the message key. Click Download Value to download the message content.

Related operations

View the details of a message retrieval task

After you enable the message retrieval feature for a topic, a message retrieval task is automatically created. You can view the details of the task, such as the topics and groups that are automatically created on the ApsaraMQ for Kafka instance, and the names of the instance and tables that are automatically created in Tablestore. You can also navigate to the details page of the Tablestore table from the Task Details page.

View consumption details

After you enable the message retrieval feature for a topic, you can view the consumption progress of the active groups in each partition of the topic. This helps you obtain information about message consumption and accumulation.

Suspend a message retrieval task

1. On the Message Retrieval page, find the topic that you want to manage and choose More > Suspend in the Actions column.

2. In the message that appears, click OK.

Resume a message retrieval task

You can resume a suspended message retrieval task based on your business requirements.

1. On the Message Retrieval page, find the topic that you want to manage and choose More > Enable in the Actions column.

2. In the message that appears, click OK.

Delete a message retrieval task

After you delete a message retrieval task for a topic, the relevant Tablestore table and search index are also deleted. The topic no longer provides the message retrieval feature. If you want to use the message retrieval feature for the topic again, re-create a message retrieval task and wait for data synchronization to complete.

1. On the Message Retrieval page, find the topic that you want to manage and perform one of the following operations based on the status of the message retrieval task:

   • If the task is not in the Running or Suspended state, click Delete in the Actions column.

   • If the task is in the Running or Suspended state, choose More > Delete in the Actions column.

2. In the message that appears, click OK.

   On the Message Retrieval page, you can no longer view the topic for which the message retrieval task is deleted.