Container Service for Kubernetes:Use Terway Edge to implement container communication in complex on-premises network environment

Step 1: Configure the uplink route (edge data center to on-cloud VPC)

1. On the data center core switch, set the next hop for the VPC CIDR block (192.168.0.0/16) to the VBR.

   If an additional switch or border gateway sits at a higher level, configure routing rules there as well so that packets destined for 192.168.0.0/16 are forwarded directly to the VBR via Express Connect circuits.

2. On the VBR, set the next hop for the VPC CIDR block (192.168.0.0/16) to the TR.

3. In the TR route table, set the next hop for the VPC CIDR block (192.168.0.0/16) to the on-cloud VPC.

Step 2: Configure the downlink route (on-cloud VPC to edge data center)

1. In the VPC route table, set the next hop for the data center host CIDR block (10.0.0.0/16) and the container CIDR block (10.10.0.0/16) to the TR.

2. In the TR route table, set the next hop for 10.0.0.0/16 and 10.10.0.0/16 to the VBR.

3. On the VBR, set the next hop for 10.0.0.0/16 and 10.10.0.0/16 to the data center core switch. Traffic is carried over Express Connect circuits.

Step 3: Configure BGP advertised routes for cluster nodes

After configuring the uplink and downlink routes, the VPC and data center hosts can communicate. However, container traffic cannot yet traverse the data center: the core switch has no routes for pod subnets. Fix this by creating a BGPPeer resource that designates specific nodes as BGP speakers to advertise pod routes to the switch.

Before writing the YAML, collect the following values from your network administrator:

In this example, Node-1, Node-2, and Node-3 act as BGP speakers and announce pod routes for all nodes attached to the core switch. All nodes belonging to this switch carry the label ASN=65010.

apiVersion: network.openyurt.io/v1alpha1
kind: BGPPeer
metadata:
  name: peer
spec:
  localSpeakers:
    - Node-1
    - Node-2
    - Node-3
  localAsNumber: 65010      # ASN of the container network's self-managed domain
  peerIP: 10.0.0.1          # IP address of the core switch
  peerAsNumber: 65001       # ASN of the switch's self-managed domain
  nodeSelector: ASN=65010   # Selects all nodes with this label

Step 4: Configure the switch to accept BGP routes

1. Enable BGP on your network device. Follow the configuration commands specific to your device model.

2. Configure Node-1, Node-2, and Node-3 as BGP peers on the switch. After configuration, verify that the BGP sessions are established.

   Important

   Configure BGP graceful restart with a restart timer of at least 300 seconds. This preserves routing information and prevents network interruptions if a BGP session drops unexpectedly.

Step 5: Verify connectivity

On the data center core switch, check the BGP session status and inspect the routing table. Confirm that:

• The BGP session with each speaker node shows an Established state.

• Routes to the pod subnets appear in the routing table. For example, a route to 10.10.0.0/24 should point to the corresponding node's IP (10.0.0.10) as the next hop.

The exact commands vary by switch vendor. Refer to your device documentation for the BGP neighbor status and routing table inspection commands.

Two nodes on the same LAN

When both nodes are on the same LAN, they communicate over the Layer 2 network. For example, pods on Node-1 send packets directly to Node-2 through the Layer 2 network, which then delivers them into pods via the Node-2 host network stack.

Two nodes on different LANs

When nodes are on different LANs, Layer 2 connectivity does not exist between them. Use Layer 3 forwarding via BGP sessions to ensure connectivity.

The following example shows pods on Node-1 (LAN A) reaching pods on Node-5 (LAN B).

Before writing the YAML, collect the following values for each LAN:

1. Establish a BGP session between Node-1 and Switch A to advertise container routes in LAN A to Switch A:

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-a
   spec:
     localSpeakers:
       - Node-1
     localAsNumber: 65010      # ASN of LAN A container network
     peerIP: 10.0.10.1         # IP address of Switch A
     peerAsNumber: 65002       # ASN of Switch A
     nodeSelector: ASN=65010   # Selects all nodes with this label

2. Establish a BGP session between Node-5 and Switch B to advertise container routes in LAN B to Switch B:

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-b
   spec:
     localSpeakers:
       - Node-5
     localAsNumber: 65020      # ASN of LAN B container network
     peerIP: 10.0.20.1         # IP address of Switch B
     peerAsNumber: 65003       # ASN of Switch B
     nodeSelector: ASN=65020   # Selects all nodes with this label

3. Establish a BGP session between Switch A and Switch B so each switch learns the other's container routes. After this configuration, packet paths are:

   • Sending: Node-1 → Switch A → Switch B → Node-5

   • Receiving: Node-5 → Switch B → Switch A → Node-1

4. (Optional) If a core switch connects the on-cloud VPC hosts and containers, establish BGP sessions between Switch A and Core Switch, and between Switch B and Core Switch.