Use the Terway Edge network plugin - Container Service for Kubernetes

The Terway Edge network plugin is an underlay container network plugin for ACK Edge clusters, based on Terway and a custom Flannel route mode. You must install this plugin when you create a cluster, and it cannot be changed afterward.

Install the Terway network plugin

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click Create Kubernetes Cluster.

Click the ACK Edge tab and configure the following key network parameters.

Parameter	Description
VPC	The VPC that the cluster uses.
Network Plug-in	Select Terway-edge.
vSwitch	The CIDR blocks of the vSwitches for cluster nodes. We recommend selecting vSwitches from at least three different availability zones for high availability.
Pod vSwitch	The CIDR block of the vSwitch for pods on cloud nodes. This CIDR block can overlap with the vSwitch CIDR blocks for nodes.
EdgeContainer CIDR block	The IP address ranges for the container network on edge nodes. For more information, see Plan CIDR blocks for a cluster.
Service CIDR	The CIDR block for services. This CIDR block cannot overlap with the node and pod CIDR blocks.
Number of Pods per Node	The maximum number of pods that can be created on an edge node.

Manage network in Express Connect mode

If you want to connect your on-premises IDC to an ACK Edge cluster over a dedicated line to establish secure, stable, and high-speed private network communication between the IDC and an Alibaba Cloud VPC, using Alibaba Cloud Express Connect as an example, you must perform the following network configurations:

In the following example, the on-premises IDC host CIDR block is 172.16.0.0/16, the cloud VPC CIDR block is 192.168.0.0/16, and the edge pod network CIDR is 10.0.0.0/8. Routes from the edge or on-premises IDC to the cloud are called uplink routes. Routes from the cloud to the on-premises IDC or edge devices are called downlink routes.
Configure uplink routes on the vSwitch, gateway device, and virtual border router (VBR) in the on-premises IDC, and on the Express Connect router (ECR) or Cloud Enterprise Network (CEN) instance to access the Alibaba Cloud VPC (192.168.0.0/16). This allows uplink requests to reach the ACK Edge control plane, ECS instances, and other products in the VPC.
You must also configure downlink routes so that requests from the ACK Edge control plane, ECS instances, and containers in the cloud can reach the on-premises IDC host CIDR block (172.16.0.0/16) and the container CIDR block (10.0.0.0/8).
You must customize the Express Connect circuit configurations based on your specific scenario. For more information, see Express Connect.

Use Terway components in the cloud

In cloud node pools, the Terway Edge network plugin functions the same as the Terway component in an ACK Pro cluster. For more information, see Terway network plugin.

Note

Terway in an ACK Edge cluster supports only shared elastic network interface (ENI) mode. It does not support features such as DataPath, NetworkPolicy, or Trunk ENI.

Use Terway components on the ENS network

On Edge Node Service (ENS) nodes, the Terway Edge network plugin provides container networking based on ENS elastic network interfaces (ENIs). For more information, see Use Terway on the ENS network.

Use Flannel components at the edge

At the edge, the Terway Edge network plugin uses a self-developed Route mode of the Flannel network plugin. After a node is added, the ACK Edge cluster control plane automatically allocates a pod CIDR block to the node. Pod IP addresses on this node are assigned from this CIDR block, and container routes are configured in the host's routing table.

Advertise container routes by using BGP

When containers on two different nodes communicate, the container network packets must be forwarded through the host's network stack.

If two hosts are in the same local area network (LAN), the host routes configured by Flannel can resolve the destination host address and deliver the container network packets.
If two hosts are in different LANs, the source host forwards container network packets to an external network device. However, because the external device has no route for the container CIDR block, the packets cannot be delivered.

Flannel advertises container routes

To add container routes to external network devices, Flannel's route mode starts a BGP service. This service establishes BGP sessions with external network devices and dynamically advertises the container routes in the LAN to those devices.

Configure BGP to advertise container routes

Note

External network devices such as Layer 3 switches must support BGP and be configurable.

BGPPeer is a new CustomResourceDefinition (CRD) used to configure BGP neighbors.

Step 1: Configure BGP clients in the cluster

Copy the following content to a file named bgppeer.yaml and customize the configuration based on your requirements.

apiVersion: network.openyurt.io/v1alpha1
kind: BGPPeer
metadata:
  name: peer
spec:
  localSpeakers:
    - node-1
    - node-2
  localAsNumber: 65010
  peerIP: 172.16.0.1
  peerAsNumber: 65001
  nodeSelector: alibabacloud.com/nodepool-id=npxxx
  # Optional
  authPassword:
    secretKeyRef:
      name: bgp-secret
      key: password
---
# Optional
apiVersion: v1
kind: Secret
metadata:
  name: bgp-secret
  namespace: kube-system
type: Opaque
data:
  password: bXlTZWNyZXRWYWx1ZQ==  # The base64-encoded value.

Parameters:

Parameter	Required	Description
metadata.name	Yes	The name of the BGPPeer.
spec.localSpeakers	Yes	The BGP client nodes within the cluster. They advertise all container CIDR blocks for nodes in the LAN. For high availability, we recommend selecting at least two nodes.
spec.localAsNumber	Yes	The Autonomous System Number (ASN) of the BGPPeer. This is the ASN of the autonomous system where the BGP client resides. In BGP, an ASN uniquely identifies an autonomous system, which is typically a single LAN. The private ASN range is 64512 to 65535.
spec.peerIP	Yes	The IP address of the peer device, such as a LAN gateway or Layer 3 switch. This device establishes BGP sessions with BGP client nodes in the cluster.
spec.peerAsNumber	Yes	The ASN of the peer device's autonomous system.
spec.gateway	No	A custom gateway address for container communication across autonomous systems. By default, this is the gateway or switch address of the LAN.
spec.nodeSelector	Yes	The node selector used to select nodes that belong to the autonomous system of this BGPPeer. We recommend that you use a node pool label to select all nodes in a node pool. If all edge nodes are in the same LAN (one autonomous system), set this parameter to `all()`.
spec.authPassword	No	The password for the BGP session. To use a password, create a Kubernetes Secret in the kube-system namespace and reference its name and key here.

Run the following command to create the BGPPeer:
```
kubectl apply -f bgppeer.yaml
```

Run the following command to check the BGPPeer status:

kubectl get bgppeer peer

Expected output:

NAME   LOCALSPEAKERS         LOCALASNUMBER   PEERIP         PEERASNUMBER     AGE
peer   ["node-1","node-2"]   65010           172.16.0.1     65001            10m

Step 2: Configure BGP on external network devices

Important

We recommend that you select at least three nodes in the cluster as BGP peers. This keeps BGP sessions active during component upgrades and prevents container network outages caused by route aging.
Terway Edge enables BGP Graceful Restart by default with a 600-second restart period. Configure BGP Graceful Restart on your switch accordingly.

Start the BGP service based on your network device model and its BGP configuration method.
On the network device, configure the BGP nodes that you selected in Step 1 as BGP neighbors and make sure that the BGP session is established.

Step 3: Verify the BGP session and routes

Check the BGPPeer events.
```
kubectl describe bgppeers peer-1
```
If the event messages include FailedEstablish, the BGP session failed to establish.
Check whether the switch contains the container routes.