ApsaraMQ for RabbitMQ:Common query statements for troubleshooting

Prerequisites

The message log management feature is enabled and an index is configured. For more information, see Message log management.

Overview of common statements

• Query the list of IP addresses of producers by exchange

• Query the list of IP addresses of consumers by queue

• Query a message trace

• Query the consumption of messages in a queue

• Query the logs of a message in a dead-letter queue

This topic describes common query statements in Simple Log Service. For information about the operations and log formats, see Message log management.

Query the list of IP addresses of producers by exchange

In the search box of Simple Log Service, enter the following statement to query the list of IP addresses of producers:

* and Action : SendMessage and Code : 200 | 
select 
  split_part(ResourceName,',',2) as exchange_name, 
  split_part(ResourceName, ',', 3) as routing_key, 
  RemoteAddress as ip_port, 
  count(*) as total_send_num 
group by 
  exchange_name, routing_key, ip_port 
order by 
  total_send_num DESC 
limit 1000000 

The following figure shows a sample query result.

Query the list of IP addresses of consumers by queue

In the search box of Simple Log Service, enter the following statement to query the list of IP addresses of consumers:

* and Action : PushMessage and Code : 200 | 
select 
  InstanceId as instance_id,
  VHost as virtual_host, 
  Queue as queue_name, 
  RemoteAddress as ip_port, 
  count(*) as push_total_num 
group by 
  instance_id,virtual_host, ip_port, queue_name 
order by 
  push_total_num DESC
limit 10000000

The following figure shows a sample query result.

Query a message trace

In the search box of Simple Log Service, enter a query statement to query a message trace.

• Query a message trace by message ID

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and ResourceName:msgId=27127757-44dc-4373-afc5-f8ea12f****

  The following figure shows a sample query result.

  

  If a client subscribes to a message by calling the BasicConsume method, a message trace that records the sending and pushing of the message can be queried. In the queried message log, the value SendMessage indicates a call to the BasicPublish method on the client, and the value PushMessage indicates the pushing of the message by the broker to the client. If the client pulls the message by calling the BasicGet method, the value of the Action parameter in the message log is BasicGet instead of PushMessage.

  Note

  • If SendMessage is recorded only in one log entry but PushMessage or BasicGet is recorded in multiple log entries in the returned message log, the client may have failed to acknowledge the consumption of the message within the valid period by calling the BasicAck method. The client can acknowledge the consumption of a message only if you call the BasicAck method within the consumption timeout period after the message is first sent. If the BasicAck method is called outside the valid time, the broker determines that the client failed to consume the message and pushes the message again. For information about the consumption timeout period, see Parameters of instance retry policies.

  • If SendDlqMessage is recorded in addition to SendMessage and PushMessage, the message was sent to a dead-letter queue.

• Query whether a message is consumed

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and Queue:cycleCheckQueue**** and ConnectionId:00163efffe08281f-00004e11-0009732f-799c0af9bc4e4913-96b3**** and  ChannelId:1 and (ResourceName:deliveryTag=90 or Property:deliveryTag=90) and RemoteAddress:"/192.168.XX.XX:XXXX"            

  You can obtain the values of the ConnectionId, ChannelId, deliveryTag, and RemoteAddress parameters based on the log entry that records PushMessage. RemoteAddress specifies the IP address of the client. ConnectionId specifies the unique identifier of the connection. ChannelId specifies the unique identifier of the channel. deliveryTag specifies the unique identifier added to the message by the broker. If DeleteMessage is logged, the message is consumed by the client.

  The following figure shows a sample query result.

  Note

  If the client calls the BasicConsume method with the autoAck=false setting, the broker can acknowledge the consumption of a message and delete the message only after the client calls the BasicAck method. The client can acknowledge the consumption of a message only if you call the BasicAck method within the consumption timeout period after the message is first sent. If the BasicAck method is called outside the valid period, the broker determines that the message failed to be consumed and pushes the message to the client again. For more information, see Parameters of instance retry policies.

  • If the client calls the BasicAck method but DeleteMessage is not logged, the message failed to be consumed. In this case, you can check the difference between the time when the PushMessage method is called and the time when the BasicAck method is called. If the difference is greater than the consumption timeout period, the call to the BasicAck method is invalid.

  • If only PushMessage and DeleteMessage are logged, the client may have called the BasicAck method with the deliveryTag, multiple=true setting. This setting indicates that the broker acknowledges the consumption of all messages before deliveryTag at a time.

Query the consumption of messages in a queue

In the search box of Simple Log Service, enter query statements to query logs for the SendMessage, PushMessage or BasicGet, BasicAck, and DeleteMessage methods. Simple Log Service calculates the percentage share of each method with respect to total method calls. If the share of each method is the same or approximately equal, the messages are sent and consumed at a similar rate. This indicates that no messages or a small number of messages are accumulated in a queue.

InstanceId:amqp-cn-i7m29o3s**** and Vhost:cycle**** and Queue: cycleCheckQueue**** and (SendMessage or PushMessage or BasicAck  or DeleteMessage)

The following figure shows a sample query result.

If one of the following scenarios occurs in the queried message log, analyze the causes based on the following description:

• If DeleteMessage is not logged, the client may have called the BasicConsume or BasicGet method with the autoAck=true setting or failed to process the requests to acknowledge the message within the consumption timeout period. For information about the consumption timeout period, see Parameters of instance retry policies.

• If the share of calls to the PushMessage method is significantly smaller than the share of calls to the SendMessage method, the number of consumers that subscribe to the message may be excessively small. In this case, we recommend that you establish more connections and create more consumers.

• If the share of calls to the SendMessage method is approximately equal to the share of calls to the PushMessage method and the share of calls to the DeleteMessage method is significantly smaller than the share of calls to the SendMessage or PushMessage method, the client may have failed to process the requests to acknowledge the message within the consumption timeout period. For information about the consumption timeout period, see Parameters of instance retry policies.

• If the share of calls to the SendMessage, PushMessage, and DeleteMessage methods are approximately equal and the share of calls to the BasicAck method is small, the client may have called the BasicAck method with the multiple=true setting.

Query the logs of a message in a dead-letter queue

In the search box of Simple Log Service, enter a query statement to query the logs of a message in a dead-letter queue.

• Query the logs of a message that is sent to a dead-letter queue after the time to live (TTL) of the message ends

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=02a162ba-f842-440f-bfd4-2595dd19****

  The following figure shows a sample query result.

  

  The value SendMessage indicates that the client calls the BasicPublish method to send the message. The value SendDlqMessage indicates that the message is sent to a dead-letter queue after the TTL of the message ends.

  Note

  SendDlqMessage is logged only if a dead-letter queue is configured.

• Query the logs of a message that is sent to a dead-letter queue after the TTL of the queue ends

  1. Configure the dead-letter attribute and TTL of a queue.

     Map<String, Object> argument = new HashMap<>();
     argument.put("x-dead-letter-exchange", [exchangeName]);
     argument.put("x-dead-letter-routing-key", [routingKey]);
     argument.put("x-message-ttl", [ttl]);
     channel.queueDeclare([queueName], true, false, false, argument);

  2. In the search box of Simple Log Service, enter the following information to query logs for SendMessage and SendDlqMessage:

     InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a****

     

• If the client calls the BasicReject or BasicNack method with the requeue=false setting, enter the following information to query logs for SendDlqMessage:

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and (ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a**** or ResourceName:deliveryTag=1)

  

  In the preceding figure, the log entry for PushMessage indicates that the broker pushes the message to the client. After the message is received by the client, the client calls the BasicReject method with the requeue=false setting. If SendDlqMessage is logged, the message is sent to a dead-letter queue.