Use log queries to troubleshoot message lifecycle - ApsaraMQ for RabbitMQ

When messages fail to send, consumers stop receiving data, or queues accumulate messages, log queries in Simple Log Service (SLS) help you pinpoint the root cause. This topic provides ready-to-use query statements for the most common ApsaraMQ for RabbitMQ issues.

Prerequisites

Before you begin, make sure that you have:

Enabled the message log management feature on your ApsaraMQ for RabbitMQ instance
Configured an index for the log data

For setup instructions, see Logs.

Quick reference: symptoms and queries

Jump to the right query based on what you observe.

Symptom	Likely cause	Query to run
Messages not reaching a queue	Producer misconfiguration or routing issue	Find producer IP addresses by exchange
Consumer not receiving messages	Consumer disconnected or too few consumers	Find consumer IP addresses by queue
Duplicate message delivery	`BasicAck` not called within the timeout	Trace a message
Messages stuck in a queue	Consumption rate lower than production rate	Analyze queue consumption rates
Messages landing in a dead-letter queue	TTL expired or message rejected	Query dead-letter queue logs

Key concepts: log actions and acknowledgment

Each log entry includes an Action field that records what happened to the message. Understanding these actions is essential for interpreting query results.

Action	What it means
`SendMessage`	A producer called `BasicPublish` to publish a message to an exchange.
`PushMessage`	The broker delivered a message to a consumer subscribed via `BasicConsume`.
`BasicGet`	A consumer pulled a message from a queue (polling mode).
`BasicAck`	A consumer acknowledged successful processing of a message.
`DeleteMessage`	The broker removed a message from the queue after confirmed consumption.
`SendDlqMessage`	The broker routed a message to a dead-letter queue.

How acknowledgment works

When a consumer subscribes with autoAck=false, the broker waits for the consumer to call BasicAck before removing the message. Two rules govern this process:

Timeout enforcement: The consumer must call BasicAck within the consumption timeout period after the message is first delivered. If the timeout expires before BasicAck is called, the broker treats the message as unprocessed and redelivers it.
Batch acknowledgment: When a consumer calls BasicAck with multiple=true, the broker acknowledges all messages before the specified deliveryTag at a time.

For timeout configuration details, see Instance retry policy parameter description.

Log field reference

These fields appear in the SLS query statements throughout this topic. Use them to build custom queries.

Field	Description	Example value
`InstanceId`	ApsaraMQ for RabbitMQ instance ID	`amqp-cn-i7m29o3s****`
`VHost`	Virtual host name	`cycle****`
`Queue`	Queue name	`cycleCheckQueue****`
`Action`	Log action type (see table above)	`SendMessage`
`Code`	Response code (`200` = success)	`200`
`ResourceName`	Composite field containing exchange name, routing key, `msgId`, or `deliveryTag`	`msgId=27127757-...`
`RemoteAddress`	Client IP address and port	`"/192.168.XX.XX:XXXX"`
`ConnectionId`	Unique connection identifier	`00163efffe08281f-...`
`ChannelId`	Channel identifier within a connection	`1`
`Property`	Message properties (includes `deliveryTag`)	`deliveryTag=90`

Find producer IP addresses by exchange

Run this query to list producer IP addresses that published to an exchange, ranked by message count.

* and Action : SendMessage and Code : 200 |
select
  split_part(ResourceName,',',2) as exchange_name,
  split_part(ResourceName, ',', 3) as routing_key,
  RemoteAddress as ip_port,
  count(*) as total_send_num
group by
  exchange_name, routing_key, ip_port
order by
  total_send_num DESC
limit 1000000

The result groups each exchange-routing key combination with the corresponding producer IP addresses and message counts.

Find consumer IP addresses by queue

Run this query to list consumer IP addresses subscribed to a queue, ranked by the number of messages pushed.

* and Action : PushMessage and Code : 200 |
select
  InstanceId as instance_id,
  VHost as virtual_host,
  Queue as queue_name,
  RemoteAddress as ip_port,
  count(*) as push_total_num
group by
  instance_id,virtual_host, ip_port, queue_name
order by
  push_total_num DESC
limit 10000000

The result groups each queue with the corresponding consumer IP addresses and total messages received.

Trace a message

Trace by message ID

To see the full lifecycle of a specific message, query by its message ID:

InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and ResourceName:msgId=27127757-44dc-4373-afc5-f8ea12f****

Replace the InstanceId, VHost, and msgId values with your own.

How to read the results

A SendMessage entry indicates the producer called BasicPublish.
A PushMessage entry indicates the broker delivered the message to a consumer (subscription mode via BasicConsume).
A BasicGet entry appears instead of PushMessage when the consumer pulled the message (polling mode).

Note

If SendMessage appears once but PushMessage or BasicGet appears multiple times, the consumer likely failed to call BasicAck within the consumption timeout. The broker redelivered the message because it assumed consumption failed. For timeout details, see Instance retry policy parameter description.
If SendDlqMessage appears alongside SendMessage and PushMessage, the message was routed to a dead-letter queue. See Query dead-letter queue logs.

Check whether a message was consumed

To verify that a specific message was successfully consumed, query by connection details and deliveryTag:

InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and Queue:cycleCheckQueue**** and ConnectionId:00163efffe08281f-00004e11-0009732f-799c0af9bc4e4913-96b3**** and  ChannelId:1 and (ResourceName:deliveryTag=90 or Property:deliveryTag=90) and RemoteAddress:"/192.168.XX.XX:XXXX"

Get the values for ConnectionId, ChannelId, deliveryTag, and RemoteAddress from the PushMessage log entry for the message.

If DeleteMessage appears in the result, the message was successfully consumed and removed from the queue.

Sample result for message consumption check

Note

When the consumer uses autoAck=false, the broker deletes a message only after receiving a BasicAck call. The BasicAck must arrive within the consumption timeout period. If it arrives after the timeout, the broker treats it as invalid and redelivers the message.

Interpreting the results:

BasicAck logged but no DeleteMessage: Compare the timestamps of PushMessage and BasicAck. If the gap exceeds the consumption timeout, the BasicAck was invalid and the broker redelivered the message.
PushMessage and DeleteMessage logged but no BasicAck: The consumer likely called BasicAck with multiple=true, acknowledging all messages before the specified deliveryTag in a single call. No individual BasicAck entry appears for this message because a batch acknowledgment covered it.

Analyze queue consumption rates

Run this query to compare the distribution of SendMessage, PushMessage, BasicAck, and DeleteMessage actions for a queue. Roughly equal counts across all four actions indicate that messages are produced and consumed at a similar rate with no significant accumulation.

InstanceId:amqp-cn-i7m29o3s**** and Vhost:cycle**** and Queue: cycleCheckQueue**** and (SendMessage or PushMessage or BasicAck  or DeleteMessage)

Sample result for queue consumption analysis

Diagnosing imbalances:

What the logs show	Likely cause	Recommended action
`DeleteMessage` is missing entirely	Consumer uses `autoAck=true`, or `BasicAck` calls are timing out	If using `autoAck=false`, check whether `BasicAck` is called within the consumption timeout
`PushMessage` count is much lower than `SendMessage`	Too few consumers for the production rate	Add more connections and consumers to the queue
`SendMessage` and `PushMessage` are roughly equal, but `DeleteMessage` is much lower	Consumers receive messages but fail to acknowledge them in time	Check whether `BasicAck` is called within the consumption timeout
`SendMessage`, `PushMessage`, and `DeleteMessage` are roughly equal, but `BasicAck` count is low	Consumer uses batch acknowledgment (`multiple=true`)	No action needed -- this is expected behavior with batch `BasicAck`

Query dead-letter queue logs

A message enters a dead-letter queue when its TTL expires, the queue TTL expires, or a consumer explicitly rejects it. Each scenario produces different log patterns.

Note

SendDlqMessage only appears in logs when a dead-letter queue is configured for the source queue.

Message TTL expired

When a message exceeds its time-to-live (TTL), the broker routes it to the configured dead-letter queue. Query by message ID:

InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=02a162ba-f842-440f-bfd4-2595dd19****

Sample result for dead-letter message TTL

In the result:

SendMessage -- the producer published the message.
SendDlqMessage -- the broker moved the message to the dead-letter queue after its TTL expired.

Queue TTL expired

When the queue itself has a TTL configured, all messages expire after the specified duration. To query this scenario:

Declare a queue with dead-letter routing and TTL:

Map<String, Object> argument = new HashMap<>();
argument.put("x-dead-letter-exchange", [exchangeName]);
argument.put("x-dead-letter-routing-key", [routingKey]);
argument.put("x-message-ttl", [ttl]);
channel.queueDeclare([queueName], true, false, false, argument);

Query the logs for SendMessage and SendDlqMessage:

InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a****

Consumer rejected the message

When a consumer calls BasicReject or BasicNack with requeue=false, the broker routes the message to the dead-letter queue. Query by both msgId and deliveryTag to capture the full flow:

InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and (ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a**** or ResourceName:deliveryTag=1)

In the result:

PushMessage -- the broker delivered the message to the consumer.
The consumer called BasicReject with requeue=false.
SendDlqMessage -- the broker routed the rejected message to the dead-letter queue.

ApsaraMQ for RabbitMQ:Common query statements for troubleshooting

Prerequisites

Quick reference: symptoms and queries

Key concepts: log actions and acknowledgment

Log field reference

Find producer IP addresses by exchange

Find consumer IP addresses by queue

Trace a message

Trace by message ID

Check whether a message was consumed

Analyze queue consumption rates

Query dead-letter queue logs

Message TTL expired

Queue TTL expired

Consumer rejected the message

See also