The maximum transmission unit (MTU) of a network connection is the size of the largest packet that can be transmitted over the connection. The MTU size includes the size of IP headers and payload and excludes the size of Ethernet headers. The larger the MTU of a connection, the more data that can be passed in a single packet. This topic describes jumbo frames and how to change the MTUs of network interfaces on Elastic Compute Service (ECS) instances.
Jumbo frames
Jumbo frames are Ethernet frames that have more than 1,500 bytes of payload, the limit set by the IEEE 802.3 standard. Jumbo frames increase the payload size per packet and the percentage of the packet that is not packet overhead to provide high throughput and high network performance. Alibaba Cloud supports jumbo frames that are 8,500 bytes in size and allows you to send Ethernet frames that have 8,500 bytes of payload.
All ECS instances support the MTU of 1,500 bytes. The following 8th-generation instance families that are available for purchase also support the MTU of 8,500 bytes: g8i, c8i, r8i, g8a, c8a, r8a, g8ae, c8ae, r8ae, g8y, c8y, and r8y. For more information, see Overview of instance families.
Considerations
The MTUs of network paths and the MTUs of peer hosts are also factors that determine whether jumbo frames can be used. Packets whose size exceeds the MTUs are fragmented or dropped by Alibaba Cloud forwarding components. The MTU of a network path is the minimum of the MTUs of each hop in the path. For more information, see the Handling of packets that exceed the MTU of a network path section of this topic.
Connectivity or performance issues may occur when jumbo frames are used in the following scenarios:
When UDP or Internet Control Messages Protocol (ICMP) jumbo frames are used to access ECS instances or hosts that are associated with Server Load Balancer (SLB) instances, fragmented packets may not be forwarded as expected by the SLB instances and then may be dropped. This causes network connectivity issues. For more information, see the MTUs for cloud network services section of this topic.
When UDP or ICMP jumbo frames are used to establish communication in MTU mismatch scenarios, packets may be fragmented and network performance may be degraded. For more information, see the MTUs of network paths between hosts section and MTUs for cloud network services section of this topic.
Before you deploy your business on ECS instances that support jumbo frames, we recommend that you test whether connectivity and performance issues caused by MTU mismatch exist in the business scenario and properly specify the packet size to minimize performance loss.
Enable or disable jumbo frames
Use one of the following methods to enable or disable jumbo frames for an ECS instance:
Method 1:
On the ECS instance buy page, you can enable or disable jumbo frames when you select an instance type that supports jumbo frames.
Method 2:
After an ECS instance is created, enable or disable jumbo frames in the Actions column of the ECS instance.
Log on to the ECS console.
In the left-side navigation pane, choose .
In the top navigation bar, select the region and resource group to which the resource belongs.
Find the instance that supports jumbo frames and choose
> Network and Security Group > Modify Jumbo Frame Configurations in the Actions column. In the Modify Jumbo Frames Configurations dialog box, enable or disable jumbo frames.
Method 3:
Enable or disable jumbo frames by calling the ModifyInstanceAttribute operation.
After you use Method 2 or Method 3 to enable or disable jumbo frames for an ECS instance, you must perform the following steps based on the operating system of the instance for the configurations to take effect:
Restart the ECS instance.
(Optional) Log on to the instance to change the MTU of the operating system. This operation is required only for Windows operating systems for which jumbo frames are enabled. For more information, see the Change the MTU values for network interfaces on ECS instances section of this topic.
NoteIn Linux, the Dynamic Host Configuration Protocol (DHCP) server automatically obtains MTUs. If you restart a Linux instance after you enable jumbo frames for the instance, 8,500 MTU is automatically enabled and 1,500 MTU is automatically disabled on the instance without the need to manually modify the MTU settings.
After you disable jumbo frames for a Windows instance and restart the instance, the standard MTU setting (1,500 MTU) is automatically enabled and takes effect. You do not need to log on to the instance to modify the MTU settings.
MTUs of network paths between hosts
By default, paths on Alibaba Cloud networks support 1,500 MTU. For ECS instances that do not support jumbo frames, consider only the MTUs of the associated network devices such as Virtual Private Network (VPN) gateways. For more information, see the MTUs for cloud network services section of this topic.
When an ECS instance that supports jumbo frames is used as a source host, the maximum MTU supported by the destination host varies based on the usage scenario. The following table describes the maximum MTUs of network paths.
Destination host | Maximum MTU |
ECS instance that has jumbo frames enabled and resides in the same VPC or a peered VPC | 8,500 |
ECS instance that has jumbo frames disabled and resides in the same VPC or a peered VPC | 1,500 |
ECS instance in a VPC that is connected by using Cloud Enterprise Network (CEN) | |
Host in an on-premises data center that is connected by using Connect Express | |
External host that can be accessed by using an elastic IP address or a public IP address |
For more information, see Overview of VPC peering connections and Manage VPC connections.
If the sizes of packets exceed the MTU of a network path, Alibaba Cloud forwarding components fragment the packets or perform negotiations based on RFC standards. This helps ensure network connectivity, but does not guarantee network communication performance. For paths that support a maximum MTU of 1,500 bytes, we recommend that you do not use jumbo frames for UDP or ICMP communication. For information about how Alibaba Cloud forwarding components handle packets that exceed the MTU of a network path, see the Handling of packets that exceed the MTU of a network path section of this topic.
MTUs for cloud network services
When you access specific network services over non-TCP protocols such as UDP and ICMP, take note of the MTU limits on the network services. Most Alibaba Cloud services do not support jumbo frames. The following table describes the maximum MTUs that are supported by specific Alibaba Cloud network services.
Cloud network service | Maximum MTU | Description |
NAT Gateway | 1,500 | None. |
Server Load Balancer, including Application Load Balancer (ALB), Network Load Balancer (NLB), and Classic Load Balancer (CLB) | ALB instances and CLB instances cannot forward fragmented packets. Set the size limit of each packet that is sent from ECS instances to 1,500 bytes for UDP or ICMP communication. | |
Other Alibaba Cloud services, such as Object Storage Service (OSS) and ApsaraDB RDS | Traffic goes through SLB services. Take note of the limits on the SLB services. | |
Global Accelerator (GA) | None. | |
Cloud Firewall (CFW) | None. | |
PrivateLink | None. | |
VPN Gateway | 1399 | For more information, see Configure MTUs. |
If the sizes of packets exceed the MTU limit on a cloud service, Alibaba Cloud forwarding components fragment the packets or perform negotiations based on RFC standards. This helps ensure network connectivity, but does not guarantee network communication performance. For cloud services that support a maximum MTU of 1,500 or 1,399 bytes, we recommend that you do not use jumbo frames for UDP or ICMP communication. For information about how Alibaba Cloud forwarding components handle packets that exceed the MTU of a cloud service, see the Handling of packets that exceed the MTU of a network path section of this topic.
Handling of packets that exceed the MTU of a network path
To ensure network connectivity, Alibaba Cloud forwarding components use one of the following features to handle the packets that exceed the MTU of a network path based on RFC standards:
TCP maximum segment size (MSS) for TCP traffic
TCP MSS limits the size of TCP segments, excluding TCP headers and IP headers, to prevent packet fragmentation. To establish a TCP connection between a source host and a destination host, both hosts advertise an MSS value. The MSS value is calculated by using the following formula: MSS = MTU - (IP header size + TCP header size). To ensure that packets can be sent and received as expected along a network path, the MSS plus TCP headers (20 bytes) plus IP headers (20 bytes per IPv4 header or 40 bytes per IPv6 header) must be less than or equal to the MTU of the network path.
If the MSS is not appropriate for the MTU of the network path, Alibaba Cloud forwarding components perform MSS clamping to lower the MSS value advertised by the source host in a Synchronize Sequence Number (SYN) packet or the MSS value advertised by the destination host in a SYN-ACK packet.
PMTUD for non-TCP traffic (UDP traffic or ICMP traffic)
For non-TCP packets, Alibaba Cloud forwarding components use the Path MTU Discovery (PMTUD) mechanism instead of TCP MSS to detect and negotiate MTUs.
When packets exceed the MTU of a network path, Alibaba Cloud forwarding components fragment the packets in which the Don't-Fragment flag is set to 0, and drop the packets in which the Don't-Fragment flag is set to 1. After a packet is dropped, the forwarding components perform operations based on the following scenarios:
If the Don't-Fragment flag is set to 1 in the IPv4 header of the packet, the forwarding components send an ICMP Fragmentation-Needed packet that indicates the MTU of the network path to the sender of the packet.
If the Don't-Fragment flag is set to 1 in the IPv6 header of the packet, the forwarding components send an ICMPv6 Packet-Too-Long packet that indicates the MTU of the network path to the sender of the packet.
PMTUD causes packet loss and helps ensure only network connectivity. Resources in the kernel and underlying forwarding components of ECS instances are consumed to construct and respond to PMTU packets and handle fragmented packets. PMTUD significantly reduces performance. To increase performance in communication scenarios when an MTU mismatch occurs, we recommend that you disable jumbo frames for ECS instances by calling the ModifyInstanceAttribute operation and restart the ECS instances for the change to take effect.
To allow an ECS instance to respond to PMTUD, make sure that ICMP inbound rules are configured in the security groups of the instance to receive ICMP packets from different forwarding components. You must also make sure that the operating system of the instance supports PMTUD and that PMTU response is enabled for the operating system. For example, set the required options in /proc/sys/net/ipv4/ip_no_pmtu_disc to 0 to enable PMTU response for Linux.
Change the MTU values for network interfaces on ECS instances
In Alibaba Cloud, specific scenarios or network devices, such as VPN gateways and traffic mirroring, have strict requirements on the sizes of packets that are sent from ECS instances. To ensure network connectivity or improve user experience, you must reduce the maximum size of packets that ECS instances can forward. You can perform the following operations to change the MTU values for network interfaces on ECS instances to meet the packet size requirements of the preceding scenarios or network devices.
When you change the MTU values of network interfaces on an ECS instance, make sure that the specified MTU values do not exceed the MTU limit at the instance level. If the specified MTU values exceed the MTU limit at the instance level, the instance may lose network connectivity. The MTU at the instance level is 8,500 bytes when jumbo frames are enabled, and 1,500 bytes when jumbo frames are disabled or when the instance does not support jumbo frames.
In Linux, DHCP servers automatically obtain MTU values except in tunneling scenarios, such as the scenarios in which VPN gateways are used. This way, you do not need to change the MTU values of network interfaces.
In Windows, DHCP servers do not automatically obtain MTUs. When you enable or disable jumbo frames for Windows instances, perform the operations to change the MTU values of network interfaces on the instances.
In the following examples, the MTU values of network interfaces are changed to 1,399 bytes on ECS instances that run different operating systems.
Windows
In this example, an ECS instance that runs Windows Server 2019 is used. To change the MTU value of a network interface on the instance, perform the following steps:
Connect to the Windows instance.
For more information, see Connect to a Windows instance by using a password or key.
Open PowerShell.
Run the following command to view the network interfaces of the instance:
Get-NetAdapterA command output similar to the following one is returned, which indicates that the ECS instance has a network interface named Ethernet.
Run the following command to view the MTU value of the Ethernet network interface:
Get-NetAdapterAdvancedProperty -Name "Ethernet"A command output similar to the following one is returned, which indicates that the MTU value of the Ethernet network interface is 1,514.
NoteIn Windows, the methods that are used to calculate MTUs vary based on the driver. In most cases, the MTU size includes the size of the 14-byte Ethernet header. Therefore, the default MTU is 1,514.
Run the following command to specify a new MTU value for a network interface:
Get-NetAdapterAdvancedProperty -Name <"Network interface name"> -RegistryKeyword "*JumboPacket" -RegistryValue <New MTU value>For example, run the following command to set the MTU value of the Ethernet network interface to 1,413. This MTU includes the 14 bytes of the Ethernet header.
Set-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket" -RegistryValue 1413After you change the MTU value of the network interface, run the
Get-NetAdapterAdvancedProperty -Name "Ethernet" -RegistryKeyword "*JumboPacket"command to check whether the new MTU value takes effect.The following figure shows a sample command output, which indicates that the MTU value of the Ethernet network interface is changed to 1,413.
Ubuntu
In this example, an ECS instance that runs Ubuntu16.04 is used. To change the MTU value of a network interface on the instance, perform the following steps:
Connect to the Linux instance.
For more information, see Connect to a Linux instance by using a password or key.
Run the following command to view the network interfaces of the instance and the MTU values of the network interfaces:
ifconfigA command output similar to the following one is returned, which indicates that the MTU value of the eth0 network interface is 1,500.
Specify a new MTU value for a network interface.
For example, run the following commands to set the MTU value of the eth0 network interface to 1,399:
mtu=1399 ifconfig eth0 mtu ${mtu} echo "/sbin/ifconfig eth0 mtu ${mtu}" > /etc/network/if-up.d/setmtu && chmod a+x /etc/network/if-up.d/setmtuAfter you change the MTU value of the network interface, run the
ifconfigcommand to check whether the new MTU value takes effect.The following figure shows a sample command output, which indicates that the MTU value of the eth0 network interface is changed to 1,399.
CentOS
In this example, an ECS instance that runs CentOS 7.8 is used. To change the MTU value of a network interface on the instance, perform the following steps:
Connect to the Linux instance.
For more information, see Connect to a Linux instance by using a password or key.
Run the following command to view the network interfaces associated with IP addresses, such as eth0.
ifconfigA command output similar to the following one is returned, which indicates that the MTU value of the eth0 network interface is 1,500.
Specify the MTU value for a network interface.
For example, run the following commands to set the MTU value of the eth0 network interface to 1,399:
mtu=1399 echo "${mtu}" > /sys/class/net/eth0/mtu echo "echo \""${mtu}"\" > /sys/class/net/eth0/mtu">> /etc/rc.d/rc.local && chmod +x /etc/rc.d/rc.localAfter you change the MTU value of the network interface, run the
ifconfigcommand to check whether the new MTU value takes effect.The following figure shows a sample command output, which indicates that the MTU value of the eth0 network interface is changed to 1,399.
FAQ
Problem description: When you use jumbo frames on an ECS instance for UDP or ICMP traffic, the performance of the instance drops significantly.
Solution: Check whether packets are fragmented and disable jumbo frames for the instance based on your business requirements. For more information, see the Enable or disable jumbo frames section of this topic.
Problem description: When you access cloud services, such as OSS and AparaDB for RDS, over UDP or ICMP from an ECS instance for which jumbo frames are enabled, network connections cannot be established to the services.
Cause: Traffic from the ECS instance flows through SLB instances that cannot forward packet fragments. As a result, a network connection failure may occur.
Solution: To ensure that packets sent by the ECS instance to the cloud services are not fragmented, set the size limit of each packet to 1,500 bytes.
Problem description: An ECS instance cannot receive PMTU messages (ICMP error messages).
Cause: PMTU messages are dropped by the security groups of the ECS instance or throttled.
Solutions:
Check whether the security groups of the instance allow ICMP traffic. For more information, see View security group rules.
Check whether the traffic rate of the instance reaches the throttling limit.