This topic describes the scenarios of Message Queue for Apache RocketMQ so that you can better use Message Queue for Apache RocketMQ in your business.

For example, an Internet e-commerce enterprise has business covering a wide range of fields such as registration, ordering, inventory, and logistics. It also involves many business peaks such as flash sales, anniversary celebrations, and regular special offers. These activities pose great challenges to the processing performance of various microservice applications in a distributed system.

Message Queue for Apache RocketMQ[******] is an important component of the distributed system. It can cope with these challenges, for example, by implementing asynchronous decoupling for applications.

This topic first describes how Message Queue for Apache RocketMQ implements the following features in the scenario of user registration:

  • Asynchronous decoupling
  • Data consistency of distributed transactions
  • Sending and subscription of ordered messages

This topic finally describes how Message Queue for Apache RocketMQ implements load shifting and cache synchronization for large-scale servers in e-commerce enterprises' flash sales and price synchronization.

Asynchronous decoupling

Traditional processing

In one of the most common scenarios, after a user registers, the system needs to send email and SMS notifications on the registration to notify the user that the registration is successful. The traditional processing is classified into two modes:

  • Serial mode
    The following figure shows the registration process in serial mode.serial

    The data flow is as follows:

    1. The user enters the account and password and submits the registration information on the registration page. The registration information is successfully written to the registration system.
    2. Then, the registration system sends a request to the email notification system. After receiving the request, the email notification system sends an email notification to the user.
    3. After that, the email notification system sends a request to the downstream SMS notification system. After receiving the request, the SMS notification system sends an SMS notification to the user.

    After all the preceding jobs are completed, a registration result is sent to the client and the user can log on with the account.

    If each job takes 50 ms, the user needs to wait for 150 ms on the registration page before being able to log on.

  • Parallel mode
    The following figure shows the registration process in parallel mode.parallel

    The data flow is as follows:

    1. The user enters the account and password and submits the registration information on the registration page. The registration information is successfully written to the registration system.
    2. Then, the registration system separately sends a request to the email notification system and SMS notification system at the same time. After receiving the request, the email notification system and SMS notification system send an email notification and an SMS notification to the user respectively.

    After all the preceding jobs are completed, a registration result is sent to the client and the user can log on with the account.

    If each job takes 50 ms, the user needs to wait for 100 ms on the registration page before being able to log on. This is because the email notification and SMS notification are sent in parallel.

The following section describes the effect of Message Queue for Apache RocketMQ when it is used in the scenario of user registration.

Asynchronous decoupling

From the viewpoint of a user, registration is complete after the registration system stores the account information of this user and then the user can log on. The user does not immediately care about the SMS and email notifications.

In the view of the registration system, the SMS and email notifications on successful registration may not be synchronously completed. After data is written into the registration system, the registration system can put other operations to Message Queue for Apache RocketMQ and immediately return the registration result to the user. The operations will be completed by Message Queue for Apache RocketMQ asynchronously.withRMQ

The data flow is as follows:

  1. The user enters the account and password and submits the registration information on the registration page. The registration information is successfully written to the registration system.
  2. Then, the registration system sends a message to Message Queue for Apache RocketMQ. Message Queue for Apache RocketMQ[******] immediately sends a response to the registration system. The registration is completed. The user can immediately log on.
  3. The downstream email and SMS notification systems consume this registration message from Message Queue for Apache RocketMQ to send email and SMS notifications to the user. Finally, the entire registration process is completed.

In this case, the user only needs to wait for 55 ms on the registration page, during which the registration data is written into the registration system and Message Queue for Apache RocketMQ.

Asynchronous decoupling is a key feature of Message Queue for Apache RocketMQ, which is intended to reduce the response time and implement decoupling. It is applicable for operations that are time-consuming and do not require immediate (synchronous) response. The operations can be put into message queues as messages. In addition, due to the use of Message Queue for Apache RocketMQ, the message sender and receiver do not need to communicate with each other or be affected by each other, which means decoupling. The only requirement is that the message format should be kept unchanged.

Data consistency of distributed transactions

In the registration process of the registration system, the user portal is on the web page of the registration system while the notification system is in the email system. The data in the two systems must be consistent.

Processing of normal messages

As described earlier, the registration system and the email notification system implement asynchronous processing by using message queues. After registration information is written into the registration system, the registration system sends a message indicating successful registration to Message Queue for Apache RocketMQ. The email notification system consumes the registration message from Message Queue for Apache RocketMQ, and then sends an email to the user indicating successful registration.inconsistentstatus

The registration process is as follows:

  1. The registration system initiates registration.
  2. The registration system sends a message indicating successful or failed registration to Message Queue for Apache RocketMQ.

    2.1 If the message is sent successfully, the process goes to step 3.

    2.2 If the message fails to be sent, the email notification system fails to receive the message indicating successful or failed registration from Message Queue for Apache RocketMQ. In this case, the email notification system cannot send an email, which causes the final status data in the email notification system and registration system to be inconsistent.

  3. The email notification system receives the message indicating successful registration from Message Queue for Apache RocketMQ.
  4. The email notification system sends an email indicating successful registration to the user.

In this case, the systems are decoupled and the upstream system does not care about the processing result of the downstream system. However, the consistency of data cannot be ensured. How can we ensure the final consistency between the email notification system and registration system?

Processing of transactional messages

In this case, we need to use transactional messages of Message Queue for Apache RocketMQ to ensure final consistency between systems.consistentstatus

The registration process is as follows:

  1. The registration system sends a half transactional message to Message Queue for Apache RocketMQ.

    1.1 If the half transactional message is sent successfully, the process goes to step 2.

    1.2 If the half transactional message fails to be sent, the registration system does not register the user but ends the process. The final data in the registration system is consistent with that in the email notification system.

  2. The registration system starts to register the user.

    2.1 If the registration succeeds, the process goes to step 3.1.

    2.2 If the registration fails, the process goes to step 3.2.

  3. The registration system sends the status of the half transactional message to Message Queue for Apache RocketMQ.

    3.1 If the half transactional message is submitted, a message indicating successful registration is generated and the process goes to step 4.

    3.2 If the half transactional message is rolled back, no message indicating successful registration is generated and the process ends. The final data in the registration system is consistent with that in the email notification system.

  4. The email notification system receives the message indicating successful registration from Message Queue for Apache RocketMQ.
  5. The email notification system sends an email indicating successful registration to the user. The final data in the registration system is consistent with that in the email notification system.

For more information about distributed transactional messages, see Transactional messages.

Sending and subscription of ordered messages

Message Queue for Apache RocketMQOrdered messages are classified into two types:

  • Globally ordered messages: All messages under a specified topic are published and consumed in strict first-in-first-out (FIFO) order.
  • Partitionally ordered messages: All messages under a specified topic are partitioned by the sharding key. Messages in one shard are published and consumed strictly in FIFO order. This ensures that one message is consumed by one process.

    In the scenario of user registration, you can use the user ID as the sharding key for partitioning. Registration messages that are created, updated, or deleted in the same shard must be published and consumed in FIFO order.

For more information about ordered messages, see Ordered messages.

Load shifting

Load shifting is also a common scenario of Message Queue for Apache RocketMQ and is widely used in flash sales or group-buying activities.

In a flash sale or group-buying activity, traffic increases sharply due to large numbers of user requests. After the flash sale application processes huge access traffic, the downstream notification system cannot bear the massive calls. In this case, certain notifications may be missed due to problems such as system breakdown. To solve these problems, Message Queue for Apache RocketMQ can be added between the application and downstream notification system.Load shifting

The flash sales process is as follows:

  1. Users initiate massive requests to the flash sales processing system.
  2. The flash sales processing system sends requests that meet the flash sales conditions to Message Queue for Apache RocketMQ according to the flash sales processing logic.
  3. The downstream notification system consumes the flash sales messages from Message Queue for Apache RocketMQ, and then sends messages indicating successful purchase to the users.
  4. The users receive the notifications on successful purchase.

Cache synchronization of large-scale servers

During the Double 11 Shopping Festival, each venue has a wide variety of items, whose prices may change in real time. In this case, the cache technology cannot meet the requirements for accessing product prices and the network interface controllers (NICs) of cache servers are fully loaded. Multiple queries of product prices may slow down the opening of venue web pages.

In this case, a broadcasting mechanism is required. One message would have been consumed by only one server in a cluster, but in the broadcasting consumption mode of Message Queue for Apache RocketMQ, the message will be consumed by all the consumers in the consumer cluster. This is similar to the price information being synchronized to all servers, which replaces the cache feature.

For more information about broadcasting consumption, see Clustering consumption and broadcasting consumption.