Container Service for Kubernetes:Use Terway Edge to implement container communication in complex on-premises network environment

Prerequisites

• An ACK Edge cluster that runs Kubernetes 1.28 or later is created.

• The cluster uses the Terway Edge network plug-in.

Billing

• For billing of Cloud Enterprise Network (CEN), see Billing rules.

• For billing of leased line (Express Connect), see Billing overview.

Communication between edge containers

Edge communication provides bidirectional communication between computing devices, containers, cloud services on the cloud and devices at the edge. You can use ACK Edge clusters to establish communication between on-cloud devices and edge devices.

• On-cloud devices are on-cloud node pools, such as Elastic Compute Service (ECS) instances, Elastic Container Instance, and other cloud services, and the managed control plane in the virtual private cloud (VPC).

• Edge devices are edge node pools, such as on-premises network devices and computing devices in the self-managed data center.

The following figure shows the communication between edge containers:

apiVersion: network.openyurt.io/v1alpha1
kind: BGPPeer
metadata:
  name: peer
spec:
  localSpeakers:
    - Node-1
    - Node-2
    - Node-3
  localAsNumber: 65010      # The ID of the self-managed domain of the container network. 
  peerIP: 10.0.0.1          # The IP address of the switch. 
  peerAsNumber: 65001       # The ID of the self-managed domain of the Switch. 
  nodeSelector: ASN=65010   # Select all nodes that have this label.

Communication between containers across LANs

If your data center has a complex network topology and multiple local area networks (LANs) are deployed in the same domain, you must connect the devices on multiple LANs to one ACK Edge cluster for centralized management. See the following figure for an example.

Containers on the same node communicate with each other via the host network stack.

The following section describes the route configuration of the Terway Edge plug-in for communication between containers across nodes:

• If the two nodes are located in the same LAN and use the same autonomous system number (ASN), the next hop of the container route is set to the corresponding node address.

• If the two nodes are on different LANs and use different ASNs, the default route of the host is used, which points to the switch address.

Two nodes on the same LAN

In scenarios in which two nodes are located in the same LAN, the nodes communicate with each other on the Layer 2 network. In the preceding figure, pods on Node-1 access pods on Node-2. Container network packets pass through the Layer 2 network to reach Node-2 and then enter the pods through the Node-2 host network stack.

Two nodes on different LANs

In scenarios in which two nodes are located in different LANs, the Layer 2 networks between the nodes do not communicate with each other. Therefore, you must use the Layer 3 forwarding capability of switches. Configure BGP sessions to ensure smooth network connections. In the preceding figure, pods on Node-1 access pods on Node-5. This example shows how to publish a container route to a network device outside the cluster. In this example, the following configurations are used.

1. Establish a BGP session between Node-1 and Switch A to publish the container routes in LAN A to Switch A. Sample YAML file:

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-a
   spec:
     localSpeakers:
       - Node-1
     localAsNumber: 65010      # The ID of the self-managed domain of the container network. 
     peerIP: 10.0.10.1         # The IP address of the switch. 
     peerAsNumber: 65002       # The ID of the self-managed domain of the Switch. 
     nodeSelector: ASN=65010   # Select all nodes that have this label.

2. Build a BGP session between Node-5 and Switch B to publish the container routes in LAN B to Switch B. Sample YAML file:

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-b
   spec:
     localSpeakers:
       - Node-5
     localAsNumber: 65020      # The ID of the self-managed domain of the container network. 
     peerIP: 10.0.20.1         # The IP address of the switch. 
     peerAsNumber: 65003       # The ID of the self-managed domain of the Switch. 
     nodeSelector: ASN=65020   # Select all nodes that have this label.

3. Establish a BGP session between Switch A and Switch B. This way, Switch A and Switch B can learn each other's container routes.

   • The route for sending packets: Node-1 -> Switch A -> Switch B -> Node-5

   • The route for receiving packets: Node-5 -> Switch B -> Switch A -> Node-1

4. (Optional) If your network device has a core switch that connects the hosts and containers of the on-cloud VPC, as shown in the following figure, you must establish a BGP session among Switch A, Core Switch, and Switch B. Establish a BGP session between Switch A and Core Switch, and establish a BGP session between Switch B and Core Switch.

Communication between containers across domains

The following example shows how to connect data centers in different network domains to ACK Edge clusters.

1. The following YAML code provides an example on how to create a BGP session to advertise routes from containers in network domain A to Switch A.

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-a
   spec:
     localSpeakers:
       - Node-1
     localAsNumber: 65010      # The ID of the self-managed domain of the container network. 
     peerIP: 10.0.10.1         # The IP address of the switch. 
     peerAsNumber: 65001       # The ID of the self-managed domain of the Switch. 
     nodeSelector: ASN=65010   # Select all nodes that have this label.

2. The following YAML code provides an example on how to create a BGP session to advertise routes from containers in network domain B to Switch B.

   apiVersion: network.openyurt.io/v1alpha1
   kind: BGPPeer
   metadata:
     name: peer-b
   spec:
     localSpeakers:
       - Node-5
     localAsNumber: 65020      # The ID of the self-managed domain of the container network. 
     peerIP: 10.0.20.1         # The IP address of the switch. 
     peerAsNumber: 65002       # The ID of the self-managed domain of the Switch. 
     nodeSelector: ASN=65020   # Select all nodes that have this label.

3. Connect different data centers.

   • If two data centers communicate with each other at the network layer via Express Connect circuits, you must establish a BGP session between Switch A and Switch B to advertise the container routes of their respective network domains to each other.

   • If two data centers do not communicate with each other at the network layer via Express Connect circuits, you must establish the BGP protocol between Switch A and the VBR, and between Switch B and the VBR. This allows you to use the VBRs to forward cross-domain network packets.

Reference

• Terway Edge

• Usage guide for Terway Edge