Configure routes for ENIs - Elastic Compute Service - Alibaba Cloud Documentation Center

Configure route tables and routing rules for ENIs to control how ECS instances send and receive traffic across multiple network interfaces.

Configure policy-based routes for ENIs

Policy-based routing forwards traffic through specified network interfaces based on pre-defined rules, instead of relying solely on default routes. This enables fine-grained traffic control across multiple ENIs.

Use cases

Communication in a multi-ENI environment: When secondary ENIs are used with EIPs or NAT gateways, reply traffic may exit through the primary ENI because its default route has higher priority. Policy-based routes enforce the source in-source out principle so that traffic enters and exits through the same ENI.
Traffic load balancing: Distribute outbound traffic across multiple ENIs by defining routing rules, when default routes do not meet your requirements.
Access control: Restrict traffic to specific interfaces or paths based on source addresses, destination addresses, or other parameters for network isolation.

Prerequisites

ENIs are bound to an ECS instance.

See the Bind an ENI to an ECS instance section of the "Create and manage ENIs" topic.
The ENIs are in effect in the operating system of the ECS instance.

See the Configure ENIs to take effect in an ECS instance section of the "Create and manage ENIs" topic.

Procedure

The following examples use a secondary ENI named eth1. Replace ENI identifiers and IP addresses with your actual values.

Linux instance

Create a route table for an ENI and add a route rule:

ip -4 route add default via <Gateway of eth1> dev eth1 table 1001
ip -4 rule add from <IP address of eth1> lookup 1001

For example, create a route table named 1001 for eth1 and add a rule for packets from 172.16.20.193 to route through eth1:

ip -4 route add default via 172.16.20.253 dev eth1 table 1001
ip -4 rule add from 172.16.20.193 lookup 1001

Persist the route configuration across reboots.

Add the route command to the instance startup configuration to persist it. Otherwise, the route is lost after a restart. Skip this step if you are only testing.
1. Open /etc/rc.local:
```
vim /etc/rc.local
```
2. Press i to enter Insert mode, add the command from the previous step, then press Esc. Enter :wq and press Enter to save and close the file.
  
  Note
  Replace the network interface identifier and gateway address with your actual values.
3. Grant execute permissions on /etc/rc.local:
```
sudo chmod +x /etc/rc.local
```

Verify that the route table and rule are created:

ip route list table 1001 && \
ip rule list

The following output confirms that the route table and rule are configured:

root@xxx:/etc/netplan# ip route list table 1001 && ip rule list
default via 172.16.20.253 dev eth1
0:	from all lookup local
32765:	from 172.16.20.193 lookup 1001
32766:	from all lookup main
32767:	from all lookup default

Windows instance

Create a policy-based route for an ENI:

route add -p <Destination network> mask <Subnet mask> <Gateway> if <Interface index> metric <Route priority>

In this example, an ENI named Ethernet 2 is used. The following command routes all packets from 172.16.12.76 through the gateway at 172.16.12.253:

route add -p 0.0.0.0 mask 0.0.0.0 172.16.12.253 if 6 metric 1

Parameters:

-p: makes the route permanent. Without -p, the route is temporary and lost after a restart.
Destination network: 0.0.0.0 indicates the default route, used when no specific routes match.

Subnet mask and gateway: Run ipconfig to view these values.

C:\Users\Administrator>ipconfig

Windows IP 配置


以太网适配器 以太网:

   连接特定的 DNS 后缀 . . . . . . . : cn-hangzhou.ecs.internal
   本地链接 IPv6 地址. . . . . . . . : fe80::6926:88f5:e002:7443%3
   IPv4 地址 . . . . . . . . . . . . : 172.16.20.241
   子网掩码  . . . . . . . . . . . . : 255.255.255.0
   默认网关. . . . . . . . . . . . . : 172.16.20.253

以太网适配器 以太网 2:

   连接特定的 DNS 后缀 . . . . . . . : cn-hangzhou.ecs.internal
   本地链接 IPv6 地址. . . . . . . . : fe80::46bd:a7fb:362e:f79f%6
   IPv4 地址 . . . . . . . . . . . . : 172.16.12.76
   子网掩码  . . . . . . . . . . . . : 255.255.255.0
   默认网关. . . . . . . . . . . . . : 172.16.12.253

Interface index: Run netsh interface ipv4 show interfaces to view the index.

In this example, the interface index (Idx) for Ethernet 2 is 6.

C:\Users\Administrator>netsh interface ipv4 show interfaces

Idx     Met         MTU          状态                名称
---  ----------  ----------  ------------  ---------------------------
  1          75  4294967295  connected     Loopback Pseudo-Interface 1
  3          15        1500  connected     以太网
  6          15        1500  connected     以太网 2

Route priority: specified in the metric <n> format. A smaller value indicates a higher priority.

Run route print to verify the route appears in the route list.

C:\Users\Administrator>route print
===========================================================================
接口列表
  3...00 16 3e 0a d5 39 ......Red Hat VirtIO Ethernet Adapter
  6...00 16 3e 4c 88 9c ......Red Hat VirtIO Ethernet Adapter #2
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 路由表
===========================================================================
活动路由:
网络目标        网络掩码          网关            接口          跃点数
      0.0.0.0          0.0.0.0    172.16.12.253   172.16.12.76     15
      0.0.0.0          0.0.0.0    172.16.12.253   172.16.20.241     15
    127.0.0.0        255.0.0.0            在链路上       127.0.0.1    331
    127.0.0.1  255.255.255.255            在链路上       127.0.0.1    331
127.255.255.255  255.255.255.255            在链路上       127.0.0.1    331
   172.16.12.0    255.255.255.0            在链路上    172.16.12.76    271
  172.16.12.76  255.255.255.255            在链路上    172.16.12.76    271
 172.16.12.255  255.255.255.255            在链路上    172.16.12.76    271
   172.16.20.0    255.255.255.0            在链路上   172.16.20.241    271
 172.16.20.241  255.255.255.255            在链路上   172.16.20.241    271
 172.16.20.255  255.255.255.255            在链路上   172.16.20.255    271
    224.0.0.0        240.0.0.0            在链路上       127.0.0.1    331
    224.0.0.0        240.0.0.0            在链路上   172.16.20.241    271
    224.0.0.0        240.0.0.0            在链路上    172.16.12.76    271
255.255.255.255  255.255.255.255            在链路上       127.0.0.1    331
255.255.255.255  255.255.255.255            在链路上   172.16.20.241    271
255.255.255.255  255.255.255.255            在链路上    172.16.12.76    271
===========================================================================
永久路由:
  网络地址          网络掩码  网关地址  跃点数
      0.0.0.0          0.0.0.0    172.16.12.253       1
===========================================================================

Example

This example shows how to configure policy-based routes so that an ECS instance running Alibaba Cloud Linux 3.2 receives packets through the secondary ENI (eth1) and sends reply packets through eth1 instead of the primary ENI (eth0). If your security policies include a source IP whitelist for specific ENIs, mismatched traffic paths cause legitimate requests to be rejected. Policy-based routes resolve this issue.

Prepare the environment.
1. Create an ECS instance.
  
  See Create an instance on the Custom Launch tab.
2. Bind a secondary ENI to the ECS instance.
  
  See Bind a secondary ENI.
3. Apply for an Elastic IP Address (EIP) and associate it with the eth1 secondary ENI in NAT mode.
  
  See Associate an EIP with a secondary ENI.
  
  After the binding is complete, on the ECS instance details page, click the Elastic Network Interface tab. The status of the secondary ENI changes to Bound, and the IP Address column shows the allocated public IP address (EIP) and primary private IP address.
4. Prepare a test client.
  
  Use another ECS instance with Internet access or an on-premises computer as the test client.
5. Add an inbound rule to a security group of the ECS instance to allow access from the test client's public IP address. Then run ping <EIP> on the test client to verify connectivity to the EIP associated in Step c.
  
  See the Case 4: Allow only traffic of specific protocols to access ECS instances section of the "Guidelines for using security groups and use cases" topic.
Send packets from the test client to the ECS instance:
```
ping 47.xx.xx.109
```
Replace the IP address with the EIP associated with eth1.
Monitor ICMP packets on both eth0 and eth1 of the ECS instance.
- Capture ICMP packets on eth0:
```
tcpdump -i eth0 icmp
```
- Open a new window and capture ICMP packets on eth1:
```
tcpdump -i eth1 icmp
```

View the results.

Policy-based route not configured

Packets enter through eth1 but reply packets exit through eth0. The instance receives on eth1 and responds on eth0.

Capture packets on eth0. You can see the ICMP echo reply:

[root@xxx ~]# tcpdump -i eth0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:01:55.129356 IP iZxxx > 12.xxx: ICMP echo reply, id 2048, seq 142, length 40
14:01:56.141993 IP iZxxx > 12.xxx: ICMP echo reply, id 2048, seq 143, length 40
14:01:57.145388 IP iZxxx > 12.xxx: ICMP echo reply, id 2048, seq 144, length 40
14:01:58.163042 IP iZxxx > 12.xxx: ICMP echo reply, id 2048, seq 145, length 40

Capture packets on eth1. You can see the ICMP echo request:

[root@xxx ~]# tcpdump -i eth1 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:01:55.129339 IP 12.xxx > iZxxx: ICMP echo request, id 2048, seq 142, length 40
14:01:56.141980 IP 12.xxx > iZxxx: ICMP echo request, id 2048, seq 143, length 40
14:01:57.145376 IP 12.xxx > iZxxx: ICMP echo request, id 2048, seq 144, length 40
14:01:58.163030 IP 12.xxx > iZxxx: ICMP echo request, id 2048, seq 145, length 40

The default route of eth0 has priority 100, higher than eth1's, so packets are sent from eth0.

[root@i xxx          ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.20.253   0.0.0.0         UG    100    0        0 eth0
0.0.0.0         172.16.20.253   0.0.0.0         UG    101    0        0 eth1
172.16.20.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
172.16.20.0     0.0.0.0         255.255.255.0   U     101    0        0 eth1

Policy-based route configured

Configure policy-based routes. See the Configure policy-based routes for ENIs section of this topic.

ip -4 route add default via 172.16.20.253 dev eth1 table 1001
ip -4 rule add from 172.16.20.177 lookup 1001

Run ping <EIP> on the test client.

Monitor ICMP packets on the ENIs.

Packets now enter and exit through eth1. The instance sends and receives through the same ENI based on the source in-source-out principle.

[root@xxx ~]# tcpdump -i eth0 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

[root@xxx ~]# tcpdump -i eth1 icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:15:13.249277 IP 123.xxx.xxx.4 > iZbp1gt1xxx          Z: ICMP echo request, id 2050, seq 150, length 40
14:15:13.249293 IP iZbxxx    lh49gbqovzZ :         4: ICMP echo reply, id 2050, seq 150, length 40
14:15:14.263769 IP 123.xxx.xxx.4 > iZbp1gt1xxx          Z: ICMP echo request, id 2050, seq 151, length 40
14:15:14.263783 IP iZbxxx    lh49gbqovzZ :         4: ICMP echo reply, id 2050, seq 151, length 40
14:15:15.267143 IP 123.xxx.xxx.4 > iZbp1gt1xxx          Z: ICMP echo request, id 2050, seq 152, length 40
14:15:15.267155 IP iZbxxx    lh49gbqovzZ :         4: ICMP echo reply, id 2050, seq 152, length 40
14:15:16.275436 IP 123.xxx.xxx.4 > iZbp1gt1xxx          Z: ICMP echo request, id 2050, seq 153, length 40
14:15:16.275449 IP iZbxxx    lh49gbqovzZ :         4: ICMP echo reply, id 2050, seq 153, length 40

Configure default routes for ENIs

Default routes are automatically configured when you bind ENIs to ECS instances. However, in some OS versions earlier than Ubuntu 18, such as Ubuntu 16, default routes may not be configured for secondary ENIs, which causes network connectivity issues. Follow these steps to manually configure a default route.

In this example, Ubuntu16 and the eth1 secondary ENI are used.

View ENI information:

ip a

root@xxx:/etc/network# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:15:74:ee brd ff:ff:ff:ff:ff:ff
    inet 172.16.20.231/24 brd 172.16.20.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe15:74ee/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:46:31:f4 brd ff:ff:ff:ff:ff:ff
    inet 172.16.20.199/24 brd 172.16.20.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe46:31f4/64 scope link
       valid_lft forever preferred_lft forever

The output shows that eth1 is in effect in the operating system.

View route information:

route -n

The output shows that eth1 has only an internal route and no outbound route.

root@xxx:/etc/network# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.20.253   0.0.0.0         UG    0      0        0 eth0
172.16.20.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
172.16.20.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

Example exception

Without an outbound route, communication through eth1 fails. For example, associating an EIP with eth1 for Internet access causes connectivity loss.

root@xxx               /etc/network# ping -I eth1 www.xxx.com
PING www.xxx.com (1xxx12) from 172.16.20.199 eth1: 56(84) bytes of data.
From 172.16.20.199 icmp_seq=1 Destination Host Unreachable
From 172.16.20.199 icmp_seq=2 Destination Host Unreachable
From 172.16.20.199 icmp_seq=3 Destination Host Unreachable
From 172.16.20.199 icmp_seq=4 Destination Host Unreachable
From 172.16.20.199 icmp_seq=5 Destination Host Unreachable
From 172.16.20.199 icmp_seq=6 Destination Host Unreachable
From 172.16.20.199 icmp_seq=7 Destination Host Unreachable
From 172.16.20.199 icmp_seq=8 Destination Host Unreachable
From 172.16.20.199 icmp_seq=9 Destination Host Unreachable
From 172.16.20.199 icmp_seq=10 Destination Host Unreachable
From 172.16.20.199 icmp_seq=11 Destination Host Unreachable
From 172.16.20.199 icmp_seq=12 Destination Host Unreachable
From 172.16.20.199 icmp_seq=13 Destination Host Unreachable
From 172.16.20.199 icmp_seq=14 Destination Host Unreachable

Configure the default route for eth1:
```
ip -4 route add default via 172.16.20.253 dev eth1 metric 200
```
- -4: applies to IPv4 addresses only.
- 172.16.20.253: the gateway address of eth1.
- metric 200: the route priority. A smaller value indicates a higher priority. When multiple routes to the same destination exist, the route with the smallest metric is used.
Persist the route configuration across reboots.

Add the route command to the instance startup configuration to persist it. Otherwise, the route is lost after a restart. Skip this step if you are only testing.
1. Open /etc/rc.local:
```
vim /etc/rc.local
```
2. Press i to enter Insert mode, add the command from the previous step, then press Esc. Enter :wq and press Enter to save and close the file.
  
  Note
  Replace the network interface identifier and gateway address with your actual values.
3. Grant execute permissions on /etc/rc.local:
```
sudo chmod +x /etc/rc.local
```

View the route added for eth1:

route -n

root@ixxxxxxxxxx:/etc/network# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.20.253   0.0.0.0         UG    0      0        0 eth0
0.0.0.0         172.16.20.253   0.0.0.0         UG    200    0        0 eth1
172.16.20.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
172.16.20.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

Verify that the exception is resolved

After configuring the default route, eth1 communicates as expected.

root@xxx:~/etc/network# ping  -I eth1 www.xxx.com
PING www.xxx.com (xxx) from 172.16.20.199 eth1: 56(84) bytes of data.
64 bytes from xxx: icmp_seq=1 ttl=51 time=11.6 ms
64 bytes from xxx: icmp_seq=2 ttl=51 time=11.6 ms
64 bytes from xxx: icmp_seq=3 ttl=51 time=11.6 ms
64 bytes from xxx: icmp_seq=4 ttl=51 time=11.6 ms
^C