Currently, some models of Alibaba Cloud Elastic Compute Service (ECS) can be configured with one native Network Interface Controller (NIC) and multiple Elastic Network Interfaces (ENIs) to meet service requirements.
However, communication may fail because multiple Elastic IP Addresses (EIPs) are bound to the native NIC and ENIs of an ECS instance. In this case, we need to check the routing configuration and policy rules in the system.
By default, each Linux operating system has a routing table. You can view the routing table by running the
IP route command. The following shows an example of a routing table.
default via 172.16.127.253 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1002 172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242
The first route plays a key role and declares a default route with eth0 as the outgoing port and 172.16.127.253 as the next hop. An instance with only one NIC can work normally with such a configuration. However, for any instance with a NIC and multiple ENIs, traffic enters the instance through different ports, whereas traffic leaves the instance only through eth0.
For example, when we externally ping the ENI-bound EIP, an ICMP echo request enters the instance through its ENI, whereas the ICMP echo reply is sent by the instance through eth0. This may result in inconsistent paths for inbound and outbound traffic, causing a series of problems, such as load imbalance and traffic bottleneck on a single NIC.
A Linux operating system has 255 routing tables. Table 254 is used by default, whereas route entries can be displayed by running the route command. Other routing tables can be configured to meet custom or advanced requirements.
You can run the
ip route list table [table index] command to view the route entries in the system.
[root@xiaoling-hz-test ~]# ip route list table 254 default via 172.16.127.253 dev eth0 169.254.0.0/16 dev eth0 scope link metric 1002 172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242
The preceding code shows the route entries of the default routing table 254. In addition, you can set the table index to 0 to view all route entries in the system. This is useful when you need to cyclically traverse the custom routing tables in the system.
table==253 is the default routing table. As a result, we recommend that you use tables 1 to 252 for adding custom route entries. You can go to
/etc/iproute2/rt_tables to view details of the predefined routing tables in a Linux operating system.
The following example shows that Tables 253 to 255 have been predefined.
# # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep
Why is it necessary to maintain multiple routing tables? The answer lies in policy-based routing (PBR). Currently, we need to ensure that both inbound and outbound packets are forwarded through the same NIC, such as eth0 or eth1. The EIP requires a default route that allows an ECS instance to access the public network. However, you cannot configure the default routing table with two routes that use 0.0.0.0 as the destination network and eth0 and eth1 as the outgoing ports, respectively. To meet this requirement, you can use PBR to forward packets based on the source IP address.
In brief, you can configure two routing tables, each of which has a default route. The default routes for the two tables use eth0 and eth1 as the outgoing port, respectively. Packets can be sent based on the source IP address. For example, packets whose source IP addresses are the same as that of eth0 are forwarded based on the routing table that contains the default route with eth0 as the outgoing port.
Packets whose source IP addresses are the same as that of eth1 are forwarded based on the routing table that contains the default route with eth1 as the outgoing port. This ensures that inbound and outbound traffic can be forwarded through the same port.
In a Linux operating system, a policy is indexed by using an unsigned 32-bit integer. Theoretically, the maximum value of the index can be 4294967295. However, only a few policies are commonly used.
The following configuration example uses the image
Ubuntu 16.04 64-bit, which cannot be automatically configured in Alibaba Cloud. The parameters involved are as follows.
ECS实例网关地址：172.16.127.253 ==原生网卡信息== 名称：eth0 IP地址：172.16.116.38 绑定的弹性公网IP：47.99.42.x ==弹性网卡信息== 名称：eth1 IP地址：172.16.116.39 绑定的弹性公网IP：116.62.163.x
The Alibaba Cloud ECS console displays incomplete information about instance-related NICs. We recommend that you obtain complete NIC information by calling the DescribeNetworkInterfaces operation. If an API debugging environment is unavailable, you can use Alibaba Cloud API Explorer for debugging.
View the NICs and default route in the system, and you will find that the system configuration is incomplete.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 254 default via 172.16.127.253 dev eth0 172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
/etc/network/interfaces file and append two new lines to the file to instruct the system to automatically enable eth1 upon startup and use Dynamic Host Configuration Protocol (DHCP) for the parameter configuration of eth1.
For more information about the configuration, refer to the main interface command.
auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp auto eth1 iface eth1 inet dhcp
Save the file and run
systemctl restart networking to restart the network service. The eth1 NIC already has an IP address.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip address 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0 valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff inet 172.16.116.39/20 brd 172.16.127.255 scope global eth1 valid_lft forever preferred_lft forever
Configure PBR based on the following information.
==原生网卡== 使用路由表100，策略路由优先级索引为200 添加路由的命令（网关地址根据实际情况而定）：ip route add default via 172.16.127.253 dev eth0 tab 100 添加策略的命令（源IP根据实际情况而定）：ip rule add from 172.16.116.38 tab 100 priority 200 ==弹性网卡== 使用路由表101，策略路由优先级索引为300 添加路由的命令（网关地址根据实际情况而定）：ip route add default via 172.16.127.253 dev eth1 tab 101 添加策略的命令（源IP根据实际情况而定）：ip rule add from 172.16.116.39 tab 101 priority 300
PBR configuration is complete after executing the preceding commands. You can view the configuration results by running the related commands.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 0 default via 172.16.127.253 dev eth0 table 100 default via 172.16.127.253 dev eth1 table 101 default via 172.16.127.253 dev eth0 172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38 172.16.112.0/20 dev eth1 proto kernel scope link src 172.16.116.39 root@iZbp14bxrlofsqs3d5dw43Z:~# ip rule list 0: from all lookup local 200: from 172.16.116.38 lookup 100 300: from 172.16.116.39 lookup 101 32766: from all lookup main 32767: from all lookup default
Two routing tables, Table 100 and Table 101, and two policies with priority 200 and priority 300 are added in the system. Moreover, rules involving eth0 and eth1 are added. This ensures that inbound and outbound traffic is forwarded through the same NIC.
The IP addresses of NICs can be persistent, whereas other configurations, such as routes and policies, stand invalid after a restart. To solve this problem, you can write the preceding commands to the system profile. Theoretically, a more elegant solution is to write the commands to a specific file in the
/etc/sysconfig/network-scripts/ (for CentOS) or
/etc/network/ (for Ubuntu) directory.
However, it is impossible for us to test all scenarios. As a result, we directly write the commands to the
/etc/rc.local file with released versions neglected.
For example, we can add the four
ip route/ip rule commands to
root@iZabcdeZ:~# vim /etc/rc.local ip route add default via 172.16.127.253 dev eth0 tab 100 ip rule add from 172.16.116.38 tab 100 priority 200 ip route add default via 172.16.127.253 dev eth1 tab 101 ip rule add from 172.16.116.39 tab 101 priority 300
The following can be directly supported without manual configuration. For more information, refer to Configure an ENI on Alibaba Cloud’s Website.
CentOS 7.3 64 位 CentOS 6.8 64 位 Windows Server 2016 数据中心版 64 位 Windows Server 2012 R2 数据中心版 64 位
CentOS uses similar policies, which can be viewed by running the ip route and ip rule commands. The difference is routing table indexes and policy priorities.
Alibaba Clouder - February 4, 2019
Marketplace - February 21, 2019
AlibabaCloud_Network - September 14, 2018
Marketplace - February 21, 2019
Alibaba Developer - June 30, 2020
Alibaba Cloud New Products - August 4, 2020
Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.Learn More
An end-to-end software-defined network solution for multinational enterprises.Learn More
An online computing service that offers elastic and secure virtual cloud servers to cater all your cloud hosting needs.Learn More
Connect your business globally with our stable network anytime anywhere.Learn More