Currently, some models of Alibaba Cloud Elastic Compute Service (ECS) can be configured with one native Network Interface Controller (NIC) and multiple Elastic Network Interfaces (ENIs) to meet service requirements.
However, communication may fail because multiple Elastic IP Addresses (EIPs) are bound to the native NIC and ENIs of an ECS instance. In this case, we need to check the routing configuration and policy rules in the system.
By default, each Linux operating system has a routing table. You can view the routing table by running the route or IP route command. The following shows an example of a routing table.
default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242
The first route plays a key role and declares a default route with eth0 as the outgoing port and 172.16.127.253 as the next hop. An instance with only one NIC can work normally with such a configuration. However, for any instance with a NIC and multiple ENIs, traffic enters the instance through different ports, whereas traffic leaves the instance only through eth0.
For example, when we externally ping the ENI-bound EIP, an ICMP echo request enters the instance through its ENI, whereas the ICMP echo reply is sent by the instance through eth0. This may result in inconsistent paths for inbound and outbound traffic, causing a series of problems, such as load imbalance and traffic bottleneck on a single NIC.
A Linux operating system has 255 routing tables. Table 254 is used by default, whereas route entries can be displayed by running the route command. Other routing tables can be configured to meet custom or advanced requirements.
You can run the ip route list table [table index] command to view the route entries in the system.
[root@xiaoling-hz-test ~]# ip route list table 254
default via 172.16.127.253 dev eth0
169.254.0.0/16 dev eth0 scope link metric 1002
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.115.242
The preceding code shows the route entries of the default routing table 254. In addition, you can set the table index to 0 to view all route entries in the system. This is useful when you need to cyclically traverse the custom routing tables in the system.
Note: table==253 is the default routing table. As a result, we recommend that you use tables 1 to 252 for adding custom route entries. You can go to /etc/iproute2/rt_tables to view details of the predefined routing tables in a Linux operating system.
The following example shows that Tables 253 to 255 have been predefined.
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
Why is it necessary to maintain multiple routing tables? The answer lies in policy-based routing (PBR). Currently, we need to ensure that both inbound and outbound packets are forwarded through the same NIC, such as eth0 or eth1. The EIP requires a default route that allows an ECS instance to access the public network. However, you cannot configure the default routing table with two routes that use 0.0.0.0 as the destination network and eth0 and eth1 as the outgoing ports, respectively. To meet this requirement, you can use PBR to forward packets based on the source IP address.
In brief, you can configure two routing tables, each of which has a default route. The default routes for the two tables use eth0 and eth1 as the outgoing port, respectively. Packets can be sent based on the source IP address. For example, packets whose source IP addresses are the same as that of eth0 are forwarded based on the routing table that contains the default route with eth0 as the outgoing port.
Packets whose source IP addresses are the same as that of eth1 are forwarded based on the routing table that contains the default route with eth1 as the outgoing port. This ensures that inbound and outbound traffic can be forwarded through the same port.
In a Linux operating system, a policy is indexed by using an unsigned 32-bit integer. Theoretically, the maximum value of the index can be 4294967295. However, only a few policies are commonly used.
The following configuration example uses the image Ubuntu 16.04 64-bit, which cannot be automatically configured in Alibaba Cloud. The parameters involved are as follows.
ECS实例网关地址：172.16.127.253
==原生网卡信息==
名称：eth0
IP地址：172.16.116.38
绑定的弹性公网IP：47.99.42.x
==弹性网卡信息==
名称：eth1
IP地址：172.16.116.39
绑定的弹性公网IP：116.62.163.x
The Alibaba Cloud ECS console displays incomplete information about instance-related NICs. We recommend that you obtain complete NIC information by calling the DescribeNetworkInterfaces operation. If an API debugging environment is unavailable, you can use Alibaba Cloud API Explorer for debugging.
View the NICs and default route in the system, and you will find that the system configuration is incomplete.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 254
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
Edit the /etc/network/interfaces file and append two new lines to the file to instruct the system to automatically enable eth1 upon startup and use Dynamic Host Configuration Protocol (DHCP) for the parameter configuration of eth1.
For more information about the configuration, refer to the main interface command.
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp
auto eth1
iface eth1 inet dhcp
Save the file and run systemctl restart networking to restart the network service. The eth1 NIC already has an IP address.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:0c:72:5d brd ff:ff:ff:ff:ff:ff
inet 172.16.116.38/20 brd 172.16.127.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:10:aa:7e brd ff:ff:ff:ff:ff:ff
inet 172.16.116.39/20 brd 172.16.127.255 scope global eth1
valid_lft forever preferred_lft forever
Configure PBR based on the following information.
==原生网卡==
使用路由表100，策略路由优先级索引为200
添加路由的命令（网关地址根据实际情况而定）：ip route add default via 172.16.127.253 dev eth0 tab 100
添加策略的命令（源IP根据实际情况而定）：ip rule add from 172.16.116.38 tab 100 priority 200
==弹性网卡==
使用路由表101，策略路由优先级索引为300
添加路由的命令（网关地址根据实际情况而定）：ip route add default via 172.16.127.253 dev eth1 tab 101
添加策略的命令（源IP根据实际情况而定）：ip rule add from 172.16.116.39 tab 101 priority 300
PBR configuration is complete after executing the preceding commands. You can view the configuration results by running the related commands.
root@iZbp14bxrlofsqs3d5dw43Z:~# ip route list table 0
default via 172.16.127.253 dev eth0 table 100
default via 172.16.127.253 dev eth1 table 101
default via 172.16.127.253 dev eth0
172.16.112.0/20 dev eth0 proto kernel scope link src 172.16.116.38
172.16.112.0/20 dev eth1 proto kernel scope link src 172.16.116.39
root@iZbp14bxrlofsqs3d5dw43Z:~# ip rule list
0: from all lookup local
200: from 172.16.116.38 lookup 100
300: from 172.16.116.39 lookup 101
32766: from all lookup main
32767: from all lookup default
Two routing tables, Table 100 and Table 101, and two policies with priority 200 and priority 300 are added in the system. Moreover, rules involving eth0 and eth1 are added. This ensures that inbound and outbound traffic is forwarded through the same NIC.
The IP addresses of NICs can be persistent, whereas other configurations, such as routes and policies, stand invalid after a restart. To solve this problem, you can write the preceding commands to the system profile. Theoretically, a more elegant solution is to write the commands to a specific file in the /etc/sysconfig/network-scripts/ (for CentOS) or /etc/network/ (for Ubuntu) directory.
However, it is impossible for us to test all scenarios. As a result, we directly write the commands to the /etc/rc.local file with released versions neglected.
For example, we can add the four ip route/ip rule commands to rc.local.
root@iZabcdeZ:~# vim /etc/rc.local
ip route add default via 172.16.127.253 dev eth0 tab 100
ip rule add from 172.16.116.38 tab 100 priority 200
ip route add default via 172.16.127.253 dev eth1 tab 101
ip rule add from 172.16.116.39 tab 101 priority 300
The following can be directly supported without manual configuration. For more information, refer to Configure an ENI on Alibaba Cloud’s Website.
CentOS 7.3 64 位
CentOS 6.8 64 位
Windows Server 2016 数据中心版 64 位
Windows Server 2012 R2 数据中心版 64 位
CentOS uses similar policies, which can be viewed by running the ip route and ip rule commands. The difference is routing table indexes and policy priorities.
1 posts | 0 followers
FollowAlibaba Cloud Native - June 9, 2023
Alibaba Cloud Community - September 1, 2022
Marketplace - August 17, 2020
Alibaba Cloud Native - June 9, 2023
OpenAnolis - October 26, 2022
Alibaba Clouder - February 4, 2019
1 posts | 0 followers
Follow
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn More
Quick Starts
Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.
Learn More
Elastic IP Address (EIP)
An independent public IP resource that decouples ECS and public IP resources, allowing you to flexibly manage public IP resources.
Learn More
Accelerated Global Networking Solution for Distance Learning
Alibaba Cloud offers an accelerated global networking solution that makes distance learning just the same as in-class teaching.
Learn More