ApsaraMQ for RabbitMQ:Common troubleshooting queries

Prerequisites

You have enabled the log management feature and configured indexes.

Common queries

• Find producer IP addresses by exchange

• Find consumer IP addresses by queue

• Trace a message

• Analyze queue consumption rates

• Query dead-letter messages

This topic lists common Simple Log Service (SLS) queries. For detailed steps and log format descriptions, see log management.

Find producer IP addresses by exchange

Run the following query to list the IP addresses of producers.

* and Action : SendMessage and Code : 200 | 
select 
  split_part(ResourceName,',',2) as exchange_name, 
  split_part(ResourceName, ',', 3) as routing_key, 
  RemoteAddress as ip_port, 
  count(*) as total_send_num 
group by 
  exchange_name, routing_key, ip_port 
order by 
  total_send_num DESC 
limit 1000000 

Find consumer IP addresses by queue

Run the following query to list the IP addresses of consumers.

* and Action : PushMessage and Code : 200 | 
select 
  InstanceId as instance_id,
  VHost as virtual_host, 
  Queue as queue_name, 
  RemoteAddress as ip_port, 
  count(*) as push_total_num 
group by 
  instance_id,virtual_host, ip_port, queue_name 
order by 
  push_total_num DESC
limit 10000000

Trace a message

Run a query in the search box to trace a message.

• Trace a message by message ID

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and ResourceName:msgId=27127757-44dc-4373-afc5-f8ea12f****

  The query returns two log entries: one with an Action of SendMessage (Code: 200) and another with an Action of PushMessage (Code: 200). This indicates that the message was successfully sent and pushed.

  When a client subscribes using BasicConsume, you can trace the message from sending to server delivery. A SendMessage action corresponds to a client's BasicPublish call, and a PushMessage action corresponds to the server pushing the message to the client. If the client pulls the message using BasicGet, the Action is BasicGet instead of PushMessage.

  Note

  • If the log results contain only one SendMessage entry but multiple PushMessage or BasicGet entries, the client may have failed to acknowledge the message by calling BasicAck within the consumption timeout after the first PushMessage or BasicGet. This causes the server to assume that the consumption failed and to redeliver the message. For more information about the consumption timeout, see Instance retry policy parameter description.

  • If the log results contain SendDlqMessage logs in addition to SendMessage and PushMessage, a SendDlqMessage log indicates that the current message was moved to the dead-letter queue.

• Check whether a message was consumed

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and Queue:cycleCheckQueue**** and ConnectionId:00163efffe08281f-00004e11-0009732f-799c0af9bc4e4913-96b3**** and  ChannelId:1 and (ResourceName:deliveryTag=90 or Property:deliveryTag=90) and RemoteAddress:"/192.168.XX.XX:XXXX"            

  Obtain the values for ConnectionId, ChannelId, deliveryTag, and RemoteAddress from PushMessage logs. RemoteAddress is the client's IP address. ConnectionId is the unique identifier for the connection. ChannelId is the unique identifier for the channel. The deliveryTag is the server's unique identifier for the message. A DeleteMessage log entry indicates that the message was successfully consumed by the client.

  The query returns three log entries. The Action values include DeleteMessage and BasicAck, and the Code for each is 200. This indicates that the message was successfully consumed.

  Note

  If a client sets autoAck=false when calling BasicConsume, the client must call BasicAck to confirm that the message was consumed. The server then deletes the message. The BasicAck call must be made within the consumption timeout period that starts from the PushMessage timestamp. If the timeout period is exceeded, the server considers the consumption to have failed and redelivers the message to the client. For more information, see Instance retry policy parameter description.

  • If the client called BasicAck, but a query for message consumption status does not return a DeleteMessage log entry, consumption failed. Check the time difference between the PushMessage and BasicAck entries. If the difference is greater than the consumption timeout, the BasicAck call is invalid.

  • If you find only PushMessage and DeleteMessage entries but no BasicAck entry, the client may have used BasicAck(deliveryTag, multiple=true) to acknowledge all previous messages up to the specified deliveryTag in a single call.

Analyze queue consumption rates

Run the following query to check the distribution of SendMessage, PushMessage or BasicGet, BasicAck, and DeleteMessage actions. If the percentages of these four actions are roughly equal, the production and consumption rates are balanced, with little to no message backlog.

InstanceId:amqp-cn-i7m29o3s**** and Vhost:cycle**** and Queue: cycleCheckQueue**** and (SendMessage or PushMessage or BasicAck  or DeleteMessage)

In the Quick Analysis panel on the left side of the query results page, view the distribution of the Action field. For example, the distribution might be BasicAck 25%, DeleteMessage 25%, PushMessage 25%, and SendMessage 24%.

If you see any of the following patterns in the logs, analyze the likely cause:

• No DeleteMessage entries in the log results: This can occur if the client set autoAck=true when calling BasicConsume or BasicGet. Alternatively, all BasicAck requests from the client may have exceeded the consumption timeout, rendering the calls invalid. For more information about the consumption timeout, see Instance retry policy parameter description.

• The percentage of PushMessage entries is much lower than that of SendMessage entries: This may indicate insufficient subscribers. We recommend adding more connections and creating new consumers.

• The percentages of SendMessage and PushMessage entries are similar, but the percentage of DeleteMessage entries is much lower: This may be because many BasicAck requests exceeded the consumption timeout and became invalid. For more information about the consumption timeout, see Instance retry policy parameter description.

• The percentages of SendMessage, PushMessage, and DeleteMessage entries are roughly the same, but the percentage of BasicAck entries is much lower: Check whether your code uses BasicAck(multiple=true).

Query dead-letter messages

In the log search box, enter a query to retrieve the relevant messages.

• Message routed to a dead-letter queue due to message TTL expiration

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=02a162ba-f842-440f-bfd4-2595dd19****

  The query returns two log entries. The Code field for both is 200, which indicates that the message operations were successful.

  A SendMessage entry corresponds to your BasicPublish call to send the message. A SendDlqMessage entry indicates that the message was sent to the corresponding dead-letter queue after its TTL expired.

  Note

  This log entry is generated only if you have configured a dead-letter queue.

• Message routed to a dead-letter queue due to queue TTL expiration

  1. Set the dead-letter properties and TTL for the queue.

     Map<String, Object> argument = new HashMap<>();
     argument.put("x-dead-letter-exchange", [exchangeName]);
     argument.put("x-dead-letter-routing-key", [routingKey]);
     argument.put("x-message-ttl", [ttl]);
     channel.queueDeclare([queueName], true, false, false, argument);

  2. Enter the following query in the log search box to retrieve SendMessage and SendDlqMessage entries.

     InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a****

     The query returns two log entries: one SendDlqMessage action for queue a and one SendMessage action for queue b. Both have a Code of 200. In the Quick Analysis panel on the left, the Action field distribution shows 50% SendMessage and 50% SendDlqMessage.

• Call BasicReject or BasicNack with requeue=false

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and (ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a**** or ResourceName:deliveryTag=1)

  A PushMessage log entry indicates that the server pushed a message to the client. After receiving the message, the client calls BasicReject(requeue=false). The corresponding SendDlqMessage log entry indicates that the message was routed to the dead-letter queue.