Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog RocketMQ Message Integration: Multi-Type Business Message - Normal Message

RocketMQ Message Integration: Multi-Type Business Message - Normal Message

This article introduces the core capabilities and advantages of using Message Queue for Apache RocketMQ as a messaging solution for business integration.

Preface

Since its development about a decade ago, Message Queue for Apache RocketMQ has been proven to be a stable solution for a wide variety of businesses. Now, it is widely implemented in Alibaba Group's internal processes and used by tens of thousands of Alibaba Cloud enterprise users. As a messaging solution with financial-grade reliability, Message Queue for Apache RocketMQ has always focused on building its asynchronous communication capacity in business integration. This article introduces the core capabilities and advantages of using Message Queue for Apache RocketMQ as a messaging solution for business integration. This article also describes the use scenarios, use cases, and best practices of the normal messages feature in Message Queue for Apache RocketMQ.

A typical business integration scenario is the e-commerce scenario. As large-scale sales promotions are often carried out in the e-commerce scenario, high requirements are placed on the throughput, end-to-end latency, and load shifting of the messaging system. Against this backdrop, Message Queue for Apache RocketMQ was developed and was first used in Alibaba Group's e-commerce systems.

This article aims to provide some insights into the following questions:

  • What are the features of the messages used in core business transaction systems?
  • What are the requirements for the messaging system of such messages?
  • What are the differences between such messages and the messages used in offline analysis scenarios?

A Comparison of Business Integration and Data Integration

1

Different Goals

When designing core business architectures, we must design the business logic required by the business. An example of business integration is the e-commerce transaction scenario. In this scenario, the system may be split into multiple applications using the microservice architecture. When we integrate these applications using messages, we focus on the business logic that says the orders from the users are processed as expected.

In contrast, data integration is data-centric and focuses more on the analysis of the data generated by business integration. Data integration focus on the properties and the relationship between data instead of where data originates.

Different Focuses

The coupling between the caller and the callee increases in business integration scenarios as the business logic of an enterprise expands in scope and increases in complexity, and the system typology becomes more complex. In many cases, the upstream application of a message may be the downstream application of another message, or a service may be both a message producer and a message consumer.

However, in data integration scenarios, we focus more on the heterogeneity and diversity of data instead of the system itself. First, we extract and cleanse data from various heterogeneous data sources to perform data integration analysis and then aggregate the data into structured data or reports for further analysis.

Different Requirements on Timeliness

Business integration can be viewed as online logic or hard real-time logic. The caller and the callee must responsd in a timely manner to both synchronous and asynchronous invocations in business integration scenarios. For example, an order must be processed in milliseconds. Otherwise, the user experience is poor.

However, data integration is usually used in near real-time or even offline (non-real-time) scenarios. Users crawl data from batch processing, real-time streaming, or near real-time streaming scenarios for further analysis. The backend processing is invisible to users. This also reflects the focus difference between data integration and business integration.

Desired Capabilities in the Messaging System of Business Integration

The message queue paradigm is a major business integration mode used by enterprises. Message queues implement an asynchronous communication pattern and provide capabilities that feature low coupling, high reliability, and observability. What are the benefits of using messages in the business integration system? I will briefly describe them in the following section.

2

The preceding figure shows a typical upper-layer application system in which a message is sent from Application A to Application B. After the message is initialized or structured, the message is sent to the event channel (such as Message Queue for Apache RocketMQ and Message Queue for RabbitMQ). This channel is known as the messaging system. The message is stored in the event channel and then matches and pushes to the downstream application using the distribution components of the router. Meanwhile, the observability, O&M, and monitoring features are provided to ensure the stable running of the application system.

The desired capabilities in the messaging system of business integration can be many. Here are the four core capabilities:

3

  1. Transmission of Multiple Message Types: Support the integration of diverse business scenarios, including normal messages, scheduled messages, transactional messages, and ordered messages
  2. Enriched Route Distribution Capability: Support multiple methods for route distribution, including tag-based filtering, message property-based filtering, one-to-many distribution, and one-to-one distribution
  3. Diverse Interaction Modes: Support diverse messaging interaction modes (such as synchronous and asynchronous message sending, active and passive message consumption), as well as streaming and single responses
  4. Observability System: Support the analysis of metrics, traces, and events, including tracing for a single system, tracing across systems, metric analysis, monitoring and alerting, and the display and analysis of business events

As a typical messaging solution for business integration, Message Queue for Apache RocketMQ provides a full range of messaging features, a variety of clients, a complete observability system, and a stability assurance solution to meet the preceding demands of business integration.

This article describes one of the multiple types of messages in Message Queue for Apache RocketMQ — normal messages.

4

Working Mechanism

Feature Overview

Normal messages are the simplest yet most important type of messages in Message Queue for Apache RocketMQ. As a basic message type, normal messages provide an asynchronous communication capability that features high throughput, high scalability, and low latency. Other advanced message types are produced by adding unique features to or specifying the usage of normal messages.

The following figure shows a typical topology of normal messages. As you can see, producers first send normal messages to the Message Queue for Apache RocketMQ broker for storage. Then, the broker matches and pushes these messages to consumers based on their subscriptions.

5

Features

  1. Atomicity: No correlation exists between messages. The logic for processing messages is also isolated from each other.
  2. Scalability: The capacity and capabilities of ordered messages are scalable. Features are supported (such as multi-queue storage, horizontal partitioning, and concurrent consumption).
  3. Low Latency: The processing of normal messages features simple interaction and status, as well as millisecond-level low latency communication.

Lifecycle

The lifecycle of normal messages starts with message initialization and ends with message deletion. Understanding the lifecycle of normal messages helps us locate and troubleshoot issues.

The status of a normal message can generally be divided into the following five types:

6

  • Initialized: The message is initialized by the producer and is ready to be sent to the broker.
  • Pending Consumption: The message is sent to the broker and is visible to consumers and available for consumption.
  • Consuming: The message is processed by the consumer based on business logic. If the consumption result is not committed to the broker after a period, the message is retried.
  • Consumption Commitment: The message is processed and the consumption result is committed to the broker. The broker marks whether the message is successfully consumed. Message Queue for Apache RocketMQ allows you to retain all messages by default. Message data is not deleted immediately but logically marked by the broker as consumed. This enables consumers to backtrack and reprocess the message.
  • Deleted: Message Queue for Apache RocketMQ deletes the messages from the physical file based on the order in which they are stored in the broker.

Scenarios and Cases

I briefly introduced the working mechanism of ordered messages in the previous section. What are their typical use scenarios? Among all types of messages in Message Queue for Apache RocketMQ, normal messages are the most widely and largely used. This type of message is primarily used in scenarios (such as service decoupling and calling), as well as batch data collection and transmission.

Scenarios

1) Microservice Calling and Decoupling

  • Asynchronous Decoupling: Asynchronous invocations of microservices can be implemented using normal messages. This shortens the business flow and response time.
  • Load Shifting: Normal messages allow the accumulation of large numbers of messages to prevent system instability caused by insufficient consumption capability in the downstream application.

7

2) Real-Time Data Transmission

  • High-Throughput Transmission: Normal messages can be scaled out unlimitedly and feature data transmission with high throughput. This facilitates data collection and reporting.
  • Real-Time Transmission: Normal messages are transmitted and delivered in real-time. This way, consumers can consume messages and analyze related data in time.

8

Case Study

1) Scenario

A trading platform is an online system where people can exchange money for goods based on agreed terms and conditions. A trading process involves the interaction between multiple subsystems, such as payment, logistics, ordering, and operations. Normal messages are used for asynchronous decoupling to ensure the stability and efficiency of the trading process, especially during large-scale online sales promotions.

2) Pain Points

Complex Order State Machine: The lifecycle of orders is long, as they are forwarded between multiple downstream subsystems. Synchronous invocations can be time-consuming and lead to a bad user experience.

Huge Numbers of Orders: Huge numbers of orders need to be processed in a large-scale sales promotion scenario. The insufficient processing capability of subsystems may result in a system crash.

Persistence of Order Changes and Transactional Consistency for Downstream Invocations in Distributed Scenarios: The forwarding of order status requires that the status change of the database is consistent with downstream invocations, which is known as transactional consistency.

9

Quick Start

We learned about the use scenarios and cases for Message Queue for Apache RocketMQ in the preceding sections. Now, let's take a look at how to use the code.

Send Normal Messages

10

The process for sending normal messages is very simple. However, we must take note of the following points:

  • Complete Information Is Included during Initialization: When you initialize a normal message, you must configure (such parameters as topic, tag, key, and load). You can configure these parameters based on your business requirements.
  • Capture Results and Exceptions: If a message is sent successfully, you need to obtain the response result. If a message failed to be sent, you need to capture the exception and retry the message.

Consume Normal Messages

11

Message Queue for Apache RocketMQ supports both the active and passive consumption modes.

If you use the passive consumption mode to consume messages, all you need to do is to register a consumption listener. Then, the listener processes the consumption logic and returns the consumption result. The result of the consumption failure is required if you want Message Queue for Apache RocketMQ to retry a message. If an exception is thrown, this is also considered a returned result. When such results are returned to the broker, the consumption is completed.

Compared with the passive consumption mode, the active consumption mode is more flexible. Consumers can actively obtain messages and concurrently consume the messages based on their business requirements. The consumption results are returned to the Message Queue for Apache RocketMQ broker after the consumers finish the consumption.

Apache RocketMQ Distributed message queue AlibabaMQ Messaging Message Queue Service
0 0 0
Share on

Read previous post:

Seven Core Issues about eBPF

Read next post:

How China Everbright Bank Uses the RocketMQ-Based Distributed Messaging Platform

Alibaba Cloud Native Community

480 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

480 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All