When you ping the internal endpoint of an ApsaraDB RDS instance from an Elastic Compute Service (ECS) instance, you get Destination Host Unreachable and cannot connect to the database.
Symptom
Running ping from the ECS instance returns the following output:
PING rm-bpxxxx.mysql.rds.aliyuncs.com (172.17.0.31) 56(84) bytes of data.
From xxxx (172.17.0.1) icmp_seq=1 Destination Host Unreachable
From xxxx (172.17.0.1) icmp_seq=2 Destination Host Unreachable
From xxxx (172.17.0.1) icmp_seq=3 Destination Host UnreachableCause
The CIDR block of a service running on the ECS instance conflicts with the private IP address of the RDS instance. In most cases, the conflicting service is Docker or another container runtime.
Diagnose the conflict
Ping the internal endpoint of the RDS instance to get its private IP address. In this example, the address is
172.17.0.31.On the ECS instance, run the following command to view the routing table:
route -nExample output:
Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.17.2.253 0.0.0.0 UG 0 0 0 eth0 169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0 172.17.0.0 0.0.0.0 255.255.255.0 U 0 0 0 br-350b446c181aLook for a NIC whose CIDR block overlaps with the private IP address of the RDS instance. In this example, the
br-350b446c181abridge NIC owns the172.17.0.0/24block, which includes172.17.0.31. Traffic destined for the RDS instance is routed to the bridge instead of the network gateway, causing the error.
Solutions
Solution 1: Change the Docker bridge CIDR (when the conflict comes from Docker on the ECS instance)
This solution does not apply to RDS for SQL Server instances. Use Solution 2 instead.
Stopping Docker or changing its bridge CIDR interrupts running containers. Perform this operation during off-peak hours.
Verify that the new CIDR block does not conflict with any other services or containers on the ECS instance.
The following steps use Linux as an example.
Stop the Docker service:
sudo systemctl stop dockerOpen the Docker daemon configuration file, typically located at
/etc/docker/daemon.json:sudo vim /etc/docker/daemon.jsonSet the
bipfield to a CIDR block that does not conflict with your RDS instance. This example uses192.168.0.0/16:{ "bip": "192.168.0.0/16" }Save and close the file.
Start the Docker service:
sudo systemctl start docker
Solution 2: Change the vSwitch of the RDS instance (when you cannot modify the container service, or for SQL Server instances)
Change the vSwitch associated with the RDS instance to one whose CIDR block does not conflict with the container service.
Do not change the virtual private cloud (VPC) of the RDS instance. By default, different VPCs cannot communicate over an internal network. Changing only the vSwitch preserves internal network connectivity between the RDS instance and other Alibaba Cloud services.
Use the following guides based on your database engine: