How to use event streams to route messages between ApsaraMQ for Kafka instances - EventBridge

This topic describes how to use the event stream feature provided by EventBridge to route messages between ApsaraMQ for Kafka instances.

Prerequisites

EventBridge is activated and the required permissions are granted to a Resource Access Management (RAM) user. For more information, see Activate EventBridge and grant permissions to a RAM user.
ApsaraMQ for Kafka instances are purchased and deployed. Make sure that the instances are in the Running state. For more information, see Step2: Purchase and deploy an instance in the "Procedure" section of the Overview topic.

Background information

As a type of lightweight channel for processing end-to-end streaming data in real time, event streams allow you to filter and transform lightweight streaming data and synchronize data between data warehouses, between data processing programs, and between data analysis and data processing systems. You can use an event stream to route messages from a source ApsaraMQ for Kafka instance to a destination ApsaraMQ for Kafka instance without the need to define an event bus. For more information, see Overview.

Step 1: Create an event stream for the destination instance

Note

You must create an event stream for the destination instance. For example, if you want to route messages from an ApsaraMQ for Kafka instance that resides in the China (Beijing) region to an ApsaraMQ for Kafka instance that resides in the China (Shanghai) region, you must create an event stream in the China (Shanghai) region.

Log on to the EventBridge console.
In the top navigation bar, select a region.
In the left-side navigation pane, click Event Streams.
On the Event Streams page, click Create Event Stream.

On the Create Event Stream page, configure the Task Name and Description parameters and follow the on-screen instructions to configure other parameters. Then, click Save. The following section describes the parameters:

Task Creation

In the Source step, set the Data Provider parameter to ApsaraMQ for Kafka and follow the on-screen instructions to configure other parameters. Then, click Next Step. The following table describes the parameters.

Parameter	Description	Example
Region	The region where the source ApsaraMQ for Kafka instance resides.	China (Beijing)
ApsaraMQ for Kafka Instance	The ApsaraMQ for Kafka instance in which the messages that you want to route are produced.	MQ_INST_115964845466****_ByBeUp3p
Topic	The topic on the ApsaraMQ for Kafka instance in which the messages that you want to route are produced.	topic
Group ID	The name of the consumer group on the source instance. Use a separate consumer group to create the event source. Do not use a consumer group that is in use. Otherwise, existing messages may fail to be sent and received.	GID_http_1
Concurrency Quota (Consumers)	The number of consumers on the source instance.	1
Consumer Offset	The offset from which messages are consumed.	Latest Offset
Network Configuration	The type of the network over which you want to route messages.	Basic Network
VPC	The virtual private cloud (VPC). This parameter is required only if you set Network Configuration to Self-managed Internet.	vpc-bp17fapfdj0dwzjkd****
vSwitch	The ID of the vSwitch with which the ApsaraMQ for Kafka instance is associated. This parameter is required only if you set Network Configuration to Self-managed Internet.	vsw-bp1gbjhj53hdjdkg****
Security Group	The ID of the security group to which the ApsaraMQ for Kafka instance belongs. This parameter is required only if you set Network Configuration to Self-managed Internet.	alikafka_pre-cn-7mz2****
Data Format	The data format feature is used to encode binary data delivered from the source into a specific data format. In message routing scenarios, set this parameter to Binary.	Binary
Batch Push	The batch push feature helps you aggregate multiple events at a time. This feature is triggered if the condition that is specified by the Messages parameter or the Interval (Unit: Seconds) parameter is met. For example, if you set the Messages parameter to 100 and the Interval (Unit: Seconds) parameter to 15, the push is executed when the number of messages reaches 100 even if only 10 seconds are elapsed.	Enable
Messages	The maximum number of messages that can be sent in each function invocation. Requests are sent only when the number of messages in the backlog reaches the specified value. Valid values: 1 to 10000.	100
Interval (Unit: Seconds)	The time interval at which the function is invoked. The system sends the aggregated messages to Function Compute at the specified time interval. Valid values: 0 to 15. Unit: seconds. The value 0 specifies that messages are sent immediately after aggregation.	3

In the Filtering step, specify the rule used to filter events and click Next Step.
Note
Specific content in the message value cannot be used to filter events based on the configurations in the example. You do not need to configure the Transformation step in message routing scenarios.

In the Sink step, set the Service Type parameter to ApsaraMQ for Kafka and follow the on-screen instructions to configure other parameters. Then, click Save. The following table describes the parameters.

Parameter	Description	Example
Instance ID	The destination ApsaraMQ for Kafka instance that you created.	test
Topic	The topic on the destination ApsaraMQ for RocketMQ instance that you created.	test
Acknowledgment Mode	The mode in which the ApsaraMQ for Kafka instance sends an acknowledgment (ACK) to a client after a message is received.	None
Message Value	EventBridge extracts specific data from events, decodes the data in the Base64 format, and then routes the data to the event target.	Binary Extraction `$.data.value`
Message Key	EventBridge extracts data from an event by using JSONPath and routes the specified content of the event to the event target.	Partial Event `$.data.key`

Task Property
Specify the retry policy and dead-letter queue for the event stream. For more information, see Retry policies and dead-letter queues.

Go back to the Event Streams page, find the event stream that you created, and then click Enable in the Actions column.
In the Note message, click OK.
The event stream requires 30 to 60 seconds to be enabled. You can view the progress in the Status column on the Event Streaming page.

Step 2: Test the event stream

Log on to the ApsaraMQ for Kafka console.
In the top navigation bar, select the region where the source instance that you specified in Step 1: Create an event stream for the destination instance resides.
In the left-side navigation pane, click Instances.
On the Instances page, find the source instance that you specified in Step 1: Create an event stream for the destination instance and click Details in the Actions column.
In the left-side navigation pane, click Topics.
On the page that appears, click the name of the topic that you specified in Step 1: Create an event stream for the destination instance.
In the upper-right corner of the Topic Details page, click Send Message.
In the Start to Send and Consume Message panel, set the Method of Sending parameter to Console, configure the Message Key and Message Content parameters, and then click OK.
Go back to the Instances page.
On the Instances page, find the destination instance that you specified in Step 1: Create an event stream for the destination instance and click Details in the Actions column.
In the left-side navigation pane, click Topics.
Click the name of the topic that you specified in Step 1: Create an event stream for the destination instance.
On the Topic Details page, click the Message Query tab.
Configure the Search Method, Partition, and Point in Time parameters. Then, click Search.
Check whether the message key and value returned are the same as those of the message that you sent.