To prevent business processing exceptions caused by repeated message consumption, after receiving messages, a Message Queue for Apache RocketMQ consumer must perform idempotent processing on these messages according to the unique key defined for a specific business. This topic describes the concepts, scenarios, and processing methods of consumption idempotence.

What is consumption idempotence?

When a consumer repeatedly consumes a message, the result of repeated consumption is the same as that of one consumption, and multiple instances of consumption do not affect the business system. In this way, consumption idempotence can be implemented.

For example, a consumer is charged RMB 100 in an order for consuming a charged message. If the charged message is repeatedly delivered due to poor networks, the consumer repeatedly consumes the message, but the fee of RMB 100 is deducted only once, with only one deduction record. The fee deduction meets the requirements, and consumption idempotence is implemented in the consumption process.

Scenarios

In Internet applications, especially in the case of poor networks, Message Queue for Apache RocketMQ messages may be duplicated. If duplicate messages affect your business, perform idempotent processing on these messages.

The scenarios of message duplication are as follows:

  • Duplication during message sending

    When a network disconnection occurs or the client breaks down after a message is sent to the Message Queue for Apache RocketMQ broker and is persisted, the Message Queue for Apache RocketMQ broker fails to respond to the client. At this time, if the producer realizes that the message failed to be sent and therefore resends the message, the consumer will receive two messages subsequently with the same content and message ID.

  • Duplication during message delivery

    A message is delivered to the consumer and the related task is completed, but a network disconnection occurs when the client sends a response to the Message Queue for Apache RocketMQ broker. To ensure that the message is consumed at least once, the Message Queue for Apache RocketMQ broker will retry delivering the previously processed message after the network recovers. The consumer will subsequently receive two messages with the same content and message ID.

  • Duplication during message rebalancing (including but not limited to the cases of network jitter, broker restart, and application restart by the consumer)

    Messages are rebalanced when the Message Queue for Apache RocketMQ broker or client is restarted or resized. In this case, the consumer may receive duplicate messages.

Solution

As there may be conflicts caused by duplicate message IDs, a secure idempotent processing should not be based on message IDs. The best way is to use the unique business identifier as the key basis for idempotent processing. The unique business identifier can be set by using the message key.

For example, in a payment scenario, you can set the message key as the order number as the basis for idempotent processing. The sample code is as follows:

Message message = new Message();
message.setKey("ORDERID_100");
SendResult sendResult = producer.send(message);           

When receiving the message, the consumer can perform idempotent processing according to the message key:

consumer.subscribe("ons_test", "*", new MessageListener() {
    public Action consume(Message message, ConsumeContext context) {
        String key = message.getKey()
        // Use the unique business identifier as the key basis for idempotent processing.
    }
});