If the repeated consumption of messages affects your business, you can perform idempotent processing on the messages. This topic describes the concept, scenarios, and processing method of message idempotence.

What is message idempotence?

In mathematics and computer science, idempotent operations can be applied multiple times without changing the result beyond the initial application. In message queue services, idempotence is used to process repeated consumption of the same messages. If a consumer repeatedly consumes a message, the final consumption result is the same as the initial consumption result, and repeated consumption does not have a negative impact on the business system.

For example, a consumer deducts the payment of an order based on a payment deduction message. The amount of the payment is USD 100. The message is repeatedly delivered to the consumer due to network issues. As a result, the message is repeatedly consumed. However, the payment is deducted only once, with only one deduction record of USD 100 for the order. In this example, message idempotence is implemented in the message consumption process, and the payment deduction meets business requirements.

Scenarios

In Internet applications, especially in the case of poor networks, Message Queue for RabbitMQ messages may be repeatedly consumed. If the repeated consumption of messages affects your business, you can perform idempotent processing on the messages. Messages may be repeatedly consumed in the following scenarios:

  • A producer repeatedly sends a message to the Message Queue for RabbitMQ broker.

    After a message is sent to the broker and persisted, the network is disconnected or the client breaks down. The broker fails to respond to the client. In this case, the producer determines that the broker does not receive the message and sends the message again. As a result, the consumer receives two messages with the same content and message ID.

  • The Message Queue for RabbitMQ broker repeatedly delivers a message to a consumer.

    After a message is delivered to a consumer, the network is disconnected, and the consumer client fails to return an ACK response to the broker. In this case, the broker does not know whether the message is consumed. To ensure that the message is consumed at least once, the broker delivers the message again after the network recovers. As a result, the consumer receives two messages with the same content and message ID.

  • A message is repeatedly delivered to a consumer due to load balancing. The factors that trigger load balancing include but are not limited to network jitters, broker restart, and consumer application restart.

    Load balancing is triggered if you restart or scale the Message Queue for RabbitMQ broker or client. During load balancing, a consumer may receive repeated messages.

Idempotent processing method

You can perform the following steps to perform idempotent processing on messages by using message IDs as idempotent keys:

  1. Create a table in the database with the Message ID field used as a unique key of the table.
  2. Configure a unique message ID for each message on the producer client.
    Sample code:
    AMQP.BasicProperties props = new AMQP.BasicProperties.Builder().messageId(UUID.randomUUID().toString()).build();
channel.basicPublish("${ExchangeName}", "BindingKey", true, props, ("messageBody" + i).getBytes(StandardCharsets.UTF_8));
    For more information about message IDs, see How do I specify a message ID?
  3. Perform idempotent processing on messages based on message IDs on the consumer client.
    Sample code:
    channel.basicConsume(Producer.QueueName, false, "YourConsumerTag",
    new DefaultConsumer(channel) {
    @Override public void handleDelivery(String consumerTag, Envelope envelope,
                AMQP.BasicProperties properties, byte[] body) throws IOException {
        // 1. Obtain the unique key data of the business.
        try{
            String messageId = properties.getMessageId();
            // Message ID or other information that is used as a unique key.
            // 2. Start a database transaction.
            idempTable.insert(messageId);
            // 3. Process the received message based on the business logic.
            // 4. Commit or roll back the transaction. // If the processing is successful, the consumer client acknowledges the message. Otherwise, the consumer client does not acknowledge the message.
            channel.basicAck(envelope.getDeliveryTag(), false);
        }
        catch (PrimaryKeyException e){
            // Directly acknowledge duplicate messages.
            channel.basicAck(envelope.getDeliveryTag(), false);
        }
    }
}
);