Cloud Network Well-architected Design Guidelines:Use CEN to build an inter-region cloud network

Summary

This topic describes how to use transit routers to establish network communication among cloud networks in different regions, including virtual private clouds (VPCs), virtual border routers (VBRs), and cloud services. Inter-region network communication helps you implement data synchronization and collaboration across regions, active geo-redundancy, and geo-disaster recovery.

This topic is intended for technical engineers, including chief technology officer (CTOs), architects, developers, and operations engineers. This topic describes the solutions to building an inter-region network by using transit routers. You can reference this topic to design an inter-region network for your business.

Terms

VPC: A VPC is a custom private network that you can create on Alibaba Cloud. VPCs are logically isolated from each other at Layer 2. You can create and manage cloud service instances in your VPC, such as Elastic Compute Service (ECS), Server Load Balancer (SLB), and ApsaraDB RDS.

Express Connect: Express Connect is a networking service that connects data centers to Alibaba Cloud. You can use Express Connect to establish high-speed, reliable, and secure private connections between data centers and cloud networks. Express Connect help you improve network communication quality and security because data transmission over Express Connect is trustable and controllable.

VBRs: VBRs are an abstraction of Express Connect circuits that are isolated and virtualized by using the Layer 3 overlay and vSwitch technologies in the Software Defined Network (SDN) architecture. A VBR is deployed between the customer-premises equipment (CPE) and a VPC to exchange data between the VPC and data center.

Cloud Enterprise Network (CEN): CEN is a high availability network that runs on the global private network of Alibaba Cloud. CEN uses transit routers to establish inter-region connections between VPCs to allow VPCs to communicate with data centers and establish flexible, reliable, and enterprise-class networks in the cloud.

Cloud Data Transfer (CDT): CDT is used to bill data transfers for different Alibaba Cloud services in a centralized manner. CDT allows you to flexibly use and conveniently manage data transfer resources. This helps you effectively reduce IT costs.

• You can use CDT to bill Internet data transfers for Alibaba Cloud services that provide access to the Internet per month in each region based on tiered pricing. The unit price decreases as the amount of Internet data transfers increases.

• You can use CDT to bill inter-region data transfers for Alibaba Cloud services that provide inter-region access based on the pay-by-data-transfer metering method. CDT supports more flexible billing and you are charged for the actual amount of data transfers.

Connections between transit routers

• Directly connected transit routers can automatically learn routes from each other. You can also configure routes between the transit routers.

• Transit routers that are connected through an intermediate transit router require manual route configurations.

We recommend that you connect transit routers without using an intermediate transit router. If you use an intermediate transit router to implement access control or bandwidth multiplexing, you can connect transit routers to the intermediate transit router to establish network communication.

Inter-region bandwidth

If you connect transit routers without using an intermediate transit router, you also need to allocate bandwidth to the transit routers. Supported bandwidth allocation methods:

• Pay-by-bandwidth: Purchase a bandwidth plan of CEN for a specific area and select the pay-by-bandwidth billing method. Then, allocate the bandwidth to the regions that you want to connect. The specified bandwidth is the maximum bandwidth for connections between the regions.

• Pay-by-data-transfer: You do not need to purchase a bandwidth plan of CEN. You can select the pay-by-data-transfer billing method when you connect two transit routers and specify a maximum bandwidth value for connections between the transit routers.

Suggestions: If your business experiences large traffic fluctuations, we recommend that you use the pay-by-data-transfer billing method. If the traffic pattern of your business remains stable, we recommend that you select the pay-by-bandwidth billing method.

• If multiple services, such as online and offline services, need to use the inter-region connection, you can configure a quality of service (QoS) policy for the inter-region connection to prevent bandwidth contention.

(Optional) Security

• Routing policies of transit routers, network access control lists (ACLs) of VPCs, security groups, and firewalls apply to inter-region communication. You can configure these security measures based on your business requirements.

Stability

The underlying network for inter-region communication between transit routers is the Alibaba Cloud global transmission network. The underlay layer uses leased lines to support multiple paths and intelligent scheduling to implement disaster recovery. The overlay layer uses ZooRoute to automatically probe the availability of underlaying paths, isolate faulty paths from packet transmission, and perform disaster recovery within seconds to ensure connection high availability. We recommend that you connect transit routers without using an intermediate transit router and enable automatic route learning between the transit routers to dynamically update routes based on network topology changes.

• High reliability based on dual-zone deployment of transit routers: Transit routers can be deployed in active/standby mode. Network traffic is automatically switched between the active and standby transit routers to ensure service availability. Multiple sets of high-quality connections exist between any two nodes in the network established by CEN. When Layer 2 connections are interrupted, the network automatically converges to prevent your workloads from being interrupted.

• High reliability based on dual-zone connections to VPCs in the same region: When you connect a VPC to a transit router, select at least two zones in the region to connect the transit router to the elastic network interfaces (ENIs) of the vSwitches in the zones. This ensures high reliability for the VPC in the region. The VPC ENIs connected to the transit router are used to forward the inbound and outbound traffic of the VPC. To isolate the vSwitches from other business vSwitches and prevent private IP address waste, we recommend that you create two subnet addresses that use the /29 mask in the zones of the VPC.

• High reliability based on multi-line redundancy for inter-region connections: Inter-region connections established on transit routers rely on the infrastructure of the Alibaba Cloud transmission network. This design ensures physical line high availability for inter-region connections to maintain business continuity and increase the service uptime in the service level agreement (SLA) to 99.95%. Platinum lines will be available soon, which support an SLA up to 99.995%.

• High reliability based on hybrid deployment of Express Connect circuits and IPsec-VPN connections: For more information, see the "Reliable connections over Express Connect circuits" section in the Use Express Connect circuits to build a hybrid-cloud or multi-cloud network topic.

High performance and scalability

• High performance and scalability of transit router clusters: Each transit router supports a maximum bandwidth of 400 Gbit/s. Each VPC connection supports a maximum bandwidth of 50 Gbit/s in the China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), and Singapore regions, and 10 Gbit/s in other regions. You do not need to manually configure the specification because the bandwidth automatically scales based on demands. If you require higher performance, contact your account manager.

• Inter-region traffic scheduling: The traffic scheduling feature allows you to limit the bandwidth of inter-region connections based on differentiated services code point (DSCP) values. This feature improves network performance because each type of service can be allocated a proper amount of bandwidth resources.

• Flexible billing for inter-region bandwidth: Inter-region bandwidth can be billed on a monthly or daily basis. If Cloud Data Transfer (CDT) is activated, inter-region bandwidth of transit routers can be billed based on the pay-by-data-transfer metering method. By default, the maximum bandwidth for inter-region connections is 1 Gbit/s. You can increase the bandwidth in the Quota Center console. You can adjust the maximum bandwidth to allow inter-region bandwidth to be dynamically scaled in or scaled out. This reduces bandwidth costs for your enterprise.

• Inter-region latency: You can query the latency between two regions on the Performance > Cloud Network Performance page in the Network Intelligence Service (NIS) console. Thus, you can make informed decisions on inter-region deployment.

Security

• After you establish communication between networks, if you need to enable access control, you can configure routing policies for transit routers, ACLs for VPCs, security groups, and Cloud Firewall. Such access control measures are applicable to both intra-region communication and inter-region communication.

Observability

• Observation and analytics on inter-region traffic: You can query the volume of inter-region traffic on the Traffic Analytics > Inter-region Traffic page in the NIS console to determine the traffic status of your business. You can query traffic volumes by IP address to locate traffic anomalies. The traffic analytics feature of NIS displays the inbound and outbound traffic between VPCs across regions and between VPCs and data centers that are connected over transit routers from multiple dimensions, such as IP addresses, ports, and protocols. You can also sort the top N traffic sources.

• Monitoring and analytics on inter-region traffic: You can use the health check and basic monitoring features of CEN to monitor inter-region traffic and view the monitoring data of inter-region bandwidth of CEN and Express Connect circuits on CloudMonitor dashboards, including the outbound bandwidth, inbound bandwidth, latency, and packet loss rate.

Self-service capabilities

• We recommend that you enable alerts on NIS metrics and CloudMonitor metrics to detect anomalies in a timely manner.

• O&M engineers can activate, deploy, and configure alert rules by using infrastructure as code (IaC) tools without the need of Alibaba Cloud backend configurations. This helps you plan business development and reduce business loss.

For more information, see NIS traffic analysis.

Scenario 1: Use transit routers to establish inter-region communication between VPCs and between VPCs and data centers

Scenario: After the VPC, VBR, and VPN gateway are connected to the transit routers, the enterprise needs to create inter-region connections and allocate inter-region bandwidth to establish network communication between cloud networks and between cloud networks and data centers.

Communication between the data center and Alibaba Cloud: The enterprise uses an Express Connect circuit and an IPsec-VPN connection to connect the data center to Alibaba Cloud in the China (Hangzhou) region. To ensure connection redundancy between the data center and Express Connect access point, we recommend that you preferentially use two Express Connect circuits or one Express Connect circuit and one IPsec-VPN connection to connect your data center to Alibaba Cloud. You can configure active/standby connections to improve the reliability of hybrid-cloud communication.

Inter-region communication between cloud networks: You can create an inter-region connection between the transit routers in the China (Shanghai) and China (Hangzhou) regions. Meanwhile, you can enable CDT to bill inter-region bandwidth based on the pay-by-data-transfer metering method. This way, the VPCs in the China (Shanghai) and China (Hangzhou) regions and the data center in Hangzhou can communicate with each other.

If you need to establish network communication among three or more regions, you can create transit routers in the regions to establish inter-region connections.

Scenario 2: Use transit routers to build a full-mesh network topology across multiple regions

Scenario: The customer deployed services in VPCs in the China (Shanghai), China (Shenzhen), China (Hangzhou), and China (Beijing) regions. The customer wants to establish network communication among the regions to build a full-mesh network topology.

Multi-region communication among cloud networks: Use the transit router in each of the regions to create inter-region connections among the China (Shanghai), China (Shenzhen), China (Hangzhou), and China (Beijing) regions. We recommend that you enable the pay-by-data-transfer metering method of CDT to reduce the cost of inter-region bandwidth.

Scenario 3: Use transit routers to build a hub-spoke network topology across multiple regions

Scenario: The customer deployed services in VPCs in the China (Shanghai), China (Shenzhen), China (Hangzhou), and China (Beijing) regions. The business center is in China (Shanghai). Frontend services are deployed in the China (Shenzhen), China (Hangzhou), and China (Beijing) regions, and need to interact with the business center in China (Shanghai) in real time. The services in the China (Shenzhen), China (Hangzhou), and China (Beijing) regions do not need to communicate with each other.

Multi-region communication among cloud networks: Create an inter-region connection between the transit routers in the China (Shanghai) region and the following regions: China (Shenzhen), China (Beijing), and China (Hangzhou). We recommend that you enable the pay-by-data-transfer metering method of CDT to reduce the cost of inter-region bandwidth.

Scenario 4: Use traffic scheduling to limit bandwidth for inter-region connections

Scenario: The maximum bandwidth of a bandwidth plan that is shared by inter-region connections is a fixed value. As a result, different services may compete for bandwidth resources and some services may waste bandwidth resources. This decreases network performance. Different service traffic has different requirements for networks, as described in the following examples:

• Video conferencing and voice communication require stable networks with low latency. Packet loss and network jitter lower the communication quality.

• SaaS requires immediate response. Network congestion decreases user experience.

• File transmission requires high network throughput but is insensitive to network performance issues, such as network latency and network jitter. Sufficient bandwidth resources are required if you want to maintain high network throughput.

The traffic scheduling feature allows you to mark inter-region network traffic with DSCP values and limit the bandwidth of inter-region connections based on DSCP values. This feature improves network performance because each type of service can be allocated a proper amount of bandwidth resources.

Traffic scheduling configurations: A traffic marking policy captures network traffic based on traffic classification rules and marks the traffic with the DSCP values that you specify.

QoS policies: A QoS policy schedules network traffic to different queues based on the DSCP values that you specify for the traffic marking policies. You can specify a maximum bandwidth value for each queue to prevent services from competing for bandwidth resources.

Each QoS policy contains one default queue. The default queue is used to handle network traffic that fails to match traffic classification rules and network traffic that matches a traffic classification rule but is not scheduled to a queue. The default queue uses the remaining bandwidth resources that are not consumed by the inter-region connections. In each QoS policy, the sum of the bandwidth values of all queues cannot exceed that of the inter-region connections.

Data synchronization and collaboration across regions

Services in VPCs across regions need to communication with each other in scenarios such as data synchronization, remote O&M, and AI training.

Geo-disaster recovery

To prevent single points of failure (SPOFs) in a single city and maintain business continuity, enterprises can deploy business systems in two or more cities to build a geo-disaster recovery architecture. In addition, enterprises can use out-of-the-box Alibaba Cloud resources that are billed on a pay-as-you-go basis to implement disaster recovery at a minimum cost.

Active geo-redundancy

To prevent SPOFs in a single city and maintain business continuity, enterprises can deploy business systems in two or more cities to build an active geo-disaster recovery architecture. Meanwhile, this architecture supports nearby access that improves user experience.

Inter-region communication by using CEN and fine-grained traffic management by using QoS policies

Coding process:

1. Create a VPC in the China (Hangzhou) region and another VPC in the China (Beijing) region.

2. Create a transit router in the China (Hangzhou) region and another transit router in the China (Beijing) region.

3. Create an inter-region connection between the transit routers in the China (Hangzhou) and China (Beijing) regions. This establishes network communication between the VPCs.

4. Create a traffic marking policy and a QoS policy. The traffic scheduling feature allows you to mark inter-region network traffic between VPCs with DSCP values and limit the bandwidth of inter-region connections based on DSCP values. This feature improves network performance because each type of service can be allocated a proper amount of bandwidth resources.

Required resources:

• Two VPCs, and three vSwitches in each VPC

• Two transit routers

• One traffic marking policy

• One QoS policy

Networking between a data center and VPCs in different regions by using transit routers

Coding process:

1. Create a VPC in the China (Hangzhou) region and another VPC in the China (Beijing) region.

2. Create a transit router in the China (Hangzhou) region and another transit router in the China (Beijing) region.

3. Create two VBRs. Connect one of the VBRs to the data center in Hangzhou and another VBR to the transit router in the China (Hangzhou) region.

4. Create an inter-region connection between the transit routers in the China (Hangzhou) and China (Beijing) regions. This establishes network communication across regions.

Required resources:

• Two VPCs, and two vSwitches in each VPC

• Two VBRs

• Two transit routers