Use raw logs to analyze and locate issues

Procedure

1. Log on to the Log Service console and configure indexes for log fields.
   For more information, see Configure indexes. You must configure indexes for the common fields that are automatically generated by the system. You must also add the __tag__:__receive_time__ field, set its alias to timestamp, and then configure an index for the field, as shown in the following figure.
2. Specify a time range to query and enter a query string in the search box of Log Service to query message logs.
   Query message logs based on business requirements. For more information, see the description in the following sections:

   • Query a message trace
   • Query the consumption of messages in a queue
   • Query logs for a message in a dead-letter queue

Query a message trace

In the search box of Log Service, enter a query string to query a message trace.

• Query message logs by message ID

  In the search box of Log Service, enter a query string that contains a specified instance ID, vhost ID, and message ID to query message logs.

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and ResourceName:msgId=27127757-44dc-4373-afc5-f8ea12f****

  

  If the client subscribes to a message by BasicConsume, you can query a message trace that records how the message is pushed by the server to the client. SendMessage works in the same manner as BasicPublish. If the client calls PushMessage, the server pushes a message to the client. If the client pulls a message by BasicGet, the returned value of Action is BasicGet rather than PushMessage.

  Note

  • In the returned log results, if SendMessage is recorded only in a log entry but PushMessage or BasicGet is recorded in multiple log entries, the client may make an invalid BasicAck call. The BasicAck call is valid only if the client calls BasicAck within 60 seconds after the first PushMessage or BasicGet operation is called. BasicAck is used to confirm that a message is consumed. If the BasicAck call is invalid, the server determines that the client fails to consume the message and pushes the message again.
  • In the returned log results, you can view the log entries that record SendMessage and PushMessage. If SendDlqMessage is also logged for a message, the message is pushed to a dead-letter queue.
• Query whether a message is consumed

  InstanceId:amqp-cn-i7m29o3s**** and VHost:cycle**** and Queue:cycleCheckQueue**** and ConnectionId:00163efffe08281f-00004e11-0009732f-799c0af9bc4e4913-96b3**** and  ChannelId:1 and (ResourceName:deliveryTag=90 or Property:deliveryTag=90) and RemoteAddress:"/192.168.XX.XX:XXXX"            

  You can obtain the values of ConnectionId, ChannelId, deliveryTag, and RemoteAddress based on the log entry that records PushMessage. RemoteAddress specifies the IP address of the client. ConnectionId specifies the unique identifier of the connection. ChannelId specifies the unique identifier of the channel. deliveryTag specifies the unique identifier of the message for the server. If DeleteMessage is logged, the message is consumed by the client.

  
  Note

  • If the client calls BasicConsume with the autoAck=false setting, the server can confirm the receipt of a message and delete the message only after the client calls BasicAck. The client must call BasicAck within 60 seconds after PushMessage is called. If BasicAck is called later than 60 seconds after PushMessage is called, the server determines that the message fails to be consumed and pushes the message to the client again.
  • If BasicAck is called by the client but DeleteMessage is not logged, the message fails to be consumed. In this case, you can check the difference between the time when PushMessage is called and the time when BasicAck is called. If the difference is greater than 60 seconds, the BasicAck call is invalid.
  • If only PushMessage and DeleteMessage are logged and BasicAck is not logged, the client may call BasicAck with the multiple=true setting. The multiple=true setting indicates that the server confirms all delivery tags up to and including the tag specified in the acknowledgment at the same time.

Query the consumption of messages in a queue

In the search box of Log Service, enter query strings to query logs for SendMessage, PushMessage or BasicGet, BasicAck, and DeleteMessage. Log Service calculates the percentage of the number of times that each operation is called out of the total number of times that all operations are called. If the percentage value of each operation is the same or approximately equal, the messages are sent and consumed at a similar rate. This indicates that no messages or a small number of messages are accumulated in a queue.

InstanceId:amqp-cn-i7m29o3s**** and Vhost:cycle**** and Queue: cycleCheckQueue**** and (SendMessage or PushMessage or BasicAck  or DeleteMessage)

In the returned log results, if one of the following scenarios occurs, analyze the causes based on the following description:

• DeleteMessage is not logged. This scenario can occur if the client calls BasicConsume or BasicGet with the autoAck=true setting. This scenario can also occur if the client makes invalid BasicAck calls. The BasicAck call is valid only if the client calls BasicAck within 60 seconds after PushMessage is called.
• The percentage of the number of times that PushMessage is called is significantly smaller than the percentage of the number of times that SendMessage is called. This scenario can occur if a small number of consumers subscribe to the messages. We recommend that you establish more connections and register more consumers.
• The percentage of the number of times that SendMessage is called is approximately equal to the percentage of the number of times that PushMessage is called, but the percentage of the number of times that DeleteMessage is called is smaller by a large amount when compared to the percentage of the number of times that SendMessage or PushMessage is called. This scenario can occur if the client calls a large number of invalid BasicAck calls. The BasicAck call is valid only if the client calls BasicAck within 60 seconds after PushMessage is called.
• The percentages of the numbers of times that SendMessage, PushMessage, and DeleteMessage are called are approximately equal, but the percentage of the number of times that BasicAck is called is smaller by a large amount. This scenario can occur if the client calls BasicAck with the multiple=true setting.

Query logs for a message in a dead-letter queue

In the search box of Log Service, enter a query string to query logs for a message in a dead-letter queue.

• Query logs for a message that is sent to a dead-letter queue after the time to live (TTL) expires

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=02a162ba-f842-440f-bfd4-2595dd19****

  

  SendMessage works in the same manner as BasicPublish. The client can call SendMessage or BasicPublish to send a message. If SendDlqMessage is logged, the message that has been in a queue for a period of time that is longer than the configured TTL is sent to a dead-letter queue.

  Note SendDlqMessage is logged only after you configure a dead-letter queue.
• Configure the TTL for a message
  1. Configure the TTL for a message and the attributes that apply for a dead-letter queue.

     Map<String, Object> argument = new HashMap<>();
     argument.put("x-dead-letter-exchange", [exchangeName]);
     argument.put("x-dead-letter-routing-key", [routingKey]);
     argument.put("x-message-ttl", [ttl]);
     channel.queueDeclare([queueName], true, false, false, argument);

  2. In the search box of Log Service, enter the following information to query logs for SendMessage and SendDlqMessage:

     InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a****

     
• If the client calls BasicReject or BasicNack with the requeue=false setting, enter the following information to query logs for SendDlqMessage:

  InstanceId:amqp-cn-i7m29o3s**** and VHost:dlq**** and (ResourceName:msgId=034a75c5-d957-422f-822e-72dfad2a**** or ResourceName:deliveryTag=1)

  

  In the preceding figure, the log entry for PushMessage records that the server pushes the message to the client. After the message is received by the client, the client calls BasicReject with the requeue=false setting. If SendDlqMessage is logged, the message is sent to a dead-letter queue.