:Use MTR and analyze the MTR test results

Introduction to MTR

• MTR is a network diagnostic tool that is pre-installed on almost all Linux distributions. A Windows clone of this tool named WinMTR is provided. WinMTR is a simplified version of MTR that provides a graphical interface. WinMTR supports only some MTR parameters.

• MTR combines the functionality of the ping command and traceroute command.

• By default, the mtr command sends Internet Control Message Protocol (ICMP) packets to probe network paths. You can configure the -u parameter in Linux operating systems to use User Datagram Protocol (UDP) packets for network path probing. In Windows operating systems, WinMTR can use only ICMP packets for network path probing.

Procedure for testing a network path

The following figure shows the flowchart for testing a network path.

Steps in the flowchart:

1. Obtain the public IP address of your local network

   Access a website, such as ip.taobao.com, from a client over your local network to obtain the public IP address of the network.

2. Perform a forward path test by using the ping command and MTR

   Run ping tests and MTR tests on a client to test the path to a destination server.

   • Ping test: Repeatedly ping the domain name or IP address of the destination server from the client. We recommend that you send at least 100 packets to ping the destination server, and record the test results.

   • MTR test: Use WinMTR if the client runs Windows or run the mtr command if the client runs Linux, to perform an MTR test against the domain name or IP address of the destination server. Then, record the test results.

3. Perform a reverse path test by using the ping command and MTR

   Run ping tests and MTR tests in the operating system of the destination server to test the path to the client.

   • Ping test: Repeatedly ping the IP address of the client from the destination server. We recommend that you send at least 100 packets to ping the client, and record the test results.

   • MTR test: Use WinMTR if the destination server runs Windows or run the mtr command if the destination server runs Linux, to perform an MTR test against the IP address of the client. Then, record the test results.

4. Analyze the test results

   Analyze the test results. For information about MTR parameters, see the "Use MTR" section of this topic. After you identify the problematic node, access websites such as ip.taobao.com to query the carrier and network to which the node belongs.

   • If the problematic node resides in the local network of the client, check the local network to troubleshoot the issue.

   • If the problematic node belongs to a carrier, contact the carrier or Alibaba Cloud after-sales technical support.

Brief description of path test results

The mtrcommand provides higher accuracy. This section provides an analysis of the test results that are returned by the mtr command. The following figure shows a sample mtr command output.

Networks

In most cases, the network path from a client to a destination server passes through the following networks:

• Local network of the client

  The local network of the client consists of a LAN and the networks of local carriers,and comprises two or three nodes, as shown in section A in the preceding figure. If an exception occurs in a node in the LAN, check the LAN and troubleshoot the exception. If an exception occurs in the network of a local carrier, report the exception to the local carrier.

• Carrier networks

  The network path passes through the backbone networks of multiple carriers that comprise multiple nodes, as shown in section B in the preceding figure. If an exception occurs in a node in carrier networks, you can query the IP address of the node to identify the carrier to which the IP address belongs, and then contact the carrier or Alibaba Cloud after-sales technical support for troubleshooting.

• Local network of the destination server

  The destination server resides in the network of a carrier that comprises the two or three nodes before the IP address of the destination server on the path, as shown in section C in the preceding figure. If an exception occurs in a node in the local network of the destination server, report the exception to the local carrier.

Link load balancing

If the loads of some middle nodes on the network path are balanced as shown in section D in the preceding figure, the mtr command numbers, probes, and collects MTR data of only the start node and end node. For the other nodes, the command output indicates only the IP address or domain name information about each node.

Avg and StDev

Due to factors such as link jitters, the Wrst value and Best value of a node may differ greatly. Avg indicates the average latency of all probes after the MTR test starts, and reflects the network quality of a node. A greater StDev value indicates a larger difference between the latencies of data packets on a node and that data packets are more discrete on the node. StDev can be used to determine whether the Avg value reflects the network quality of a node. For example, if the StDev value is large, the latency of packets is unknown. Some packets may be sent at a low latency, such as 25 ms, and other packets may be sent at a high latency, such as 350 ms. In this case, the final Avg value may be in the normal range. In this scenario, the Avg value does not reflect the actual network quality.

We recommend that you analyze the Avg value and StDev value based on the following items:

• If the StDev value of a node is large, check the Best value and Wrst value of the node to determine whether a network issue occurred on the node.

• If the StDev value of a node is not large, determine whether the node experiences a network issue based on the Avg value of the node.

  Note

  No time range standards can be used to determine whether the StDev value of a node is large or not. You can determine whether the StDev value of a node is large based on the other latency values of the node. For example, if the Avg value is 30 ms and the StDev value is 25 ms, the StDev value is determined to be large. If the Avg value is 325 ms and the StDev value is 25 ms, the StDev is determined to be not large.

Loss%

If the packet loss rate (Loss%) of a node is not zero, a network exception may occur at this hop. Packet loss may occur on a node due to the following reasons:

• The ICMP transmission rate of the node is limited by the carrier for security purposes or performance reasons.

• An exception occurred in the node.

Check whether packet loss occurred on the subsequent nodes to identify the cause of packet loss.

• If no packet loss occurred on the subsequent nodes, the packet loss on the node is caused by the ICMP throttling policy of the carrier, as shown in the second hop in the preceding figure. You can ignore this packet loss issue.

• If packet loss occurred on all subsequent nodes, a network exception occurred in the node and caused packet loss, as shown in the fifth hop in the preceding figure.

• If packet loss occurred only on some of the subsequent nodes, the ICMP transmission rate of the node is limited by the carrier and a network exception occurred in the node. In this case, if packet loss repeatedly occurred on the node and the subsequent nodes and the packet loss rate of each node is different, the packet loss rate of the last several hops takes precedence. Packet loss occurred at the fifth hop, sixth hop, and seventh hop, and the packet loss rate of the seventh hop is 40%, as shown in the preceding figure. The packet loss rate of the seventh hop is used for reference.

Latency

• Latency spike at hops

  If latency spikes after a hop, a network exception occurs in the node at the hop. Latency spiked after the fifth hop, as shown in the preceding figure. A network exception occurred in the node at the hop. A high latency does not indicate that an exception exists in a node. Even though latency spiked after the fifth hop, the test data reached the destination host, as shown in the preceding figure. A high latency may also occur for the response path. We recommend that you perform a reverse path test to analyze the issue.

• Latency increase caused by ICMP throttling

  ICMP throttling on a node may also cause a latency spike on the node but does not affect the latency on the subsequent nodes. The packet loss rate at the third hop reaches 100% and a latency spike occurred at the hop, as shown in the preceding figure. The latency on the subsequent nodes immediately decreased to the normal level. You can determine that the latency spike and the packet loss on the node are caused by ICMP throttling.

Common scenarios for path exceptions

Misconfigured network of the destination host

In this example, all packets are lost at the end of data transmission, as shown in the preceding figure. ICMP may be disabled in the security policies of the destination server, such as firewall policies and iptables policies. As a result, the destination host cannot send responses, and packets cannot reach the destination IP address. You must check the security policies of the destination server.

ICMP throttling

In this example, all packets are lost at the end of data transmission, as shown in the preceding figure. ICMP may be disabled in the security policies of the destination server, such as firewall policies, iptables policies, or throttling policies of carriers. As a result, the destination host cannot send responses, and packets cannot reach the destination IP address. You must check the security policies of the destination server or perform a reverse path test to analyze the issue.

Loop

In this example, a routing loop occurred after packets passed through the fifth hop, and packets cannot reach the destination server, as shown in the preceding figure. In most cases, the routing loop is caused by an exception in the route configuration of a carrier node. You must contact the carrier to resolve the issue.

Node interruption

In this example, packets cannot receive feedback after they pass through the fourth hop. In most cases, this issue occurs due to the interruption of the node at the hop. We recommend that you perform a reverse path test to check the issue. You must contact the carrier to which the node belongs.

References

RFC792