Cloud Network Well-architected Design Guidelines:Use CEN to build a cross-region cloud network

Introduction

This guide covers how to use transit routers (TRs) to connect VPCs, VBRs, and cloud services across regions by configuring cross-region bandwidth for multi-region data synchronization, active geo-redundancy, and geo-disaster recovery.

Intended for CTOs, architects, developers, and operations engineers, this guide helps you plan and build a cross-region cloud network using TRs.

Terms

VPC: A virtual private cloud (VPC) is a logically isolated Layer 2 private network on Alibaba Cloud where you create and manage resources such as ECS, SLB, and RDS instances.

Express Connect: Express Connect establishes a high-speed, secure private channel between your data center and Alibaba Cloud for reliable data transmission.

VBR: A virtual border router (VBR) abstracts a physical Express Connect circuit port into a virtual router using Layer 3 overlay and vSwitch virtualization on an SDN architecture. VBRs route traffic between customer-premises equipment (CPE) and VPCs, bridging on-premises data centers to the cloud.

Cloud Enterprise Network: Cloud Enterprise Network (CEN) runs on Alibaba Cloud's private global network. CEN uses transit routers (TRs) to build private communication channels between VPCs across regions and between VPCs and on-premises data centers.

Cloud Data Transfer: Cloud Data Transfer (CDT) provides unified billing and settlement for cloud traffic, enabling flexible traffic management and cost reduction.

• Tiered pricing for cumulative Internet traffic of public network products, aggregated by region monthly. Higher usage yields lower unit prices.

• Pay-by-data-transfer billing for cross-region traffic, offering flexible on-demand pricing.

TR connection design

• Two directly connected TRs can automatically learn routes or be manually configured with routes for communication.

• Two TRs can be connected through an intermediate TR and manually configured with routes for communication.

Direct connection is recommended. Use an intermediate TR for transit only if you need access control or bandwidth reuse to reduce costs.

Cross-region bandwidth design

Two directly connected TRs also require bandwidth configuration. Choose one of the following methods:

• Pay-by-bandwidth: Purchase a CEN bandwidth plan for a large region and allocate bandwidth between communicating regions. The allocated bandwidth is the speed limit.

• Pay-by-data-transfer: No bandwidth plan required. Select pay-by-data-transfer when configuring the TR connection and specify a peak bandwidth (speed limit).

Recommendation: Use pay-by-data-transfer for bursty traffic and pay-by-bandwidth for stable traffic.

• For multiple services between connected regions, configure a cross-region QoS policy to prevent bandwidth contention.

Security design [Optional]

• TR routing policies, VPC network ACLs, security groups, and firewalls are all effective for cross-region access control. Design them based on your requirements.

Stability design

Cross-region TR communication runs on the Alibaba Cloud global transmission network. The underlay layer uses multi-path leased lines with intelligent disaster-recovery scheduling. The overlay layer uses ZooRoute to detect available paths and remove faulty ones, enabling failover in seconds. Use direct TR connections with automatic route learning so routes update automatically when the topology changes.

• Zone-redundant TR cluster: A TR provides primary and secondary nodes by default with automatic switchover. Multiple high-quality links exist between any two nodes. If a link is interrupted, the network converges automatically without affecting your services.

• Zone-redundant VPC connections in the same city: When associating a VPC with a TR, connect at least two vSwitch ENIs in different zones to ensure multi-zone high availability. Create two /29 subnets in the corresponding zones to isolate TR-connected ENIs from other resources and conserve IP addresses.

• Redundant cross-region connections: Each cross-region TR connection is backed by multiple physical links on the Alibaba Cloud transmission network, with an SLA of up to 99.95%. Platinum lines offer an SLA of up to 99.995%.

• Highly available hybrid cloud connections using Express Connect circuits or VPNs: Reliability design for Express Connect circuits.

High performance and elasticity design

• High performance and elasticity of TR clusters: A single TR cluster supports up to 400 Gbps forwarding. A single VPC connection supports up to 50 Gbps in the China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), and Singapore regions, and 10 Gbps in other regions. Performance scales elastically with no configuration required. For higher requirements, contact your account manager.

• Traffic scheduling for cross-region traffic: Use traffic marks to set bandwidth limits for different cross-region traffic types, ensuring sufficient bandwidth per service and improving network efficiency.

• Elastic payment for cross-region bandwidth: Pay-by-bandwidth lets you upgrade cross-region bandwidth monthly or daily. With CDT enabled, you can use pay-by-data-transfer instead. The default cross-region bandwidth limit is 1 Gbps (adjustable in Quota Center). Scale peak bandwidth up or down to optimize costs.

• Cross-region latency: Use the cloud network performance monitoring feature of Network Intelligence Service (NIS) to query cross-region latency and plan optimal multi-region deployments.

Security design

• For access control beyond basic connectivity, use TR routing policies, VPC network ACLs, security groups, or Cloud Firewall. These policies apply to both same-region and cross-region communication.

Observability design

• Cross-region network traffic analysis: Use the NIS cross-region traffic analysis feature to monitor traffic volume and detect anomalies. NIS displays inbound and outbound traffic for cross-region VPCs and on-premises data centers passing through a TR, with breakdowns by IP address, port, and protocol for top-traffic analysis.

• Cross-region network traffic monitoring: Use Cloud Monitor with CEN health check and infrastructure monitoring to view cross-region bandwidth and Express Connect circuit metrics: outbound/inbound bandwidth, latency, and packet loss rate.

Self-service design

• Enable alerting for NIS and Cloud Monitor metrics to detect threats promptly.

• O&M engineers can use Infrastructure as Code (IaC) for self-service provisioning and configuration without backend intervention from Alibaba Cloud, reducing disruption impact and accelerating development.

Network Intelligence Service (NIS): Network Traffic Analysis

Scenario 1: Connect on-premises and multi-region cloud VPCs with a TR

Scenario overview: Connect VPC, VBR, and VPN instances to a transit router, then create a cross-region connection with allocated bandwidth to link on-premises and cloud resources across regions.

IDC-to-cloud connection: The enterprise connects its IDC to Alibaba Cloud in Hangzhou using an Express Connect circuit and an IPsec-VPN connection. For redundancy, use two physical Express Connect circuits or one circuit plus one VPN connection, configured as active/standby or load-balanced.

Cross-region connection on the cloud: Create a cross-region TR connection between Shanghai and Hangzhou. Enable pay-by-data-transfer with CDT to connect the Shanghai VPC, Hangzhou VPC, and Hangzhou IDC.

If your enterprise needs to connect three or more regions, you can extend this architecture by adding more cross-region TR connections.

Scenario 2: Use TRs to build a full-mesh network across multiple regions

Scenario overview: A customer has VPCs in Shanghai, Shenzhen, Hangzhou, and Beijing that must form a full-mesh network.

Multi-region connection on the cloud: Create cross-region connections between TRs in all four regions. Enable pay-by-data-transfer with CDT to reduce bandwidth costs.

Scenario 3: Use TRs to build a hub-spoke network across multiple regions

Scenario overview: A customer has VPCs in Shanghai, Shenzhen, Hangzhou, and Beijing. Shanghai is the hub. The other regions run frontend services that interact with Shanghai in real time but do not need mutual connectivity.

Multi-region connection on the cloud: Create cross-region connections from the Shanghai TR to Shenzhen, Beijing, and Hangzhou. Enable pay-by-data-transfer with CDT to reduce costs.

Scenario 4: Use traffic scheduling to control cross-region bandwidth for different types of traffic

Scenario overview: Cross-region connection bandwidth is fixed, so different traffic types compete for bandwidth, reducing utilization and quality. Different services have different network requirements:

• Video conferencing and voice call traffic require real-time transmission. High packet loss rates and frequent jitter can degrade communication quality.

• Office Software as a Service (SaaS) traffic requires timely responses. Network congestion can degrade the user experience.

• Office file transfer traffic requires high network throughput. It needs sufficient bandwidth but has low requirements for network latency and jitter.

Traffic scheduling marks different cross-region traffic types and sets per-mark bandwidth limits, ensuring each service has sufficient bandwidth.

Traffic marking policy: A traffic marking policy uses classification rules to identify traffic and assign a DSCP value as a mark.

QoS policy: A QoS policy divides traffic into queues by DSCP value and allocates bandwidth per queue to prevent contention.

Each QoS policy includes a default queue for unmatched traffic and traffic that matches a rule but is not assigned to a queue. The default queue uses the remaining bandwidth. The sum of all queue bandwidths cannot exceed the total cross-region connection bandwidth.

Multi-region data synchronization and collaboration on the cloud

Connect VPCs across regions for data synchronization, remote O&M, and AI training.

Geo-disaster recovery

Deploy business systems across two or more city nodes for geo-disaster recovery. This prevents single-node failures from interrupting services and leverages pay-as-you-go cloud resources to minimize costs.

Active geo-redundancy

Deploy business systems across two or more city nodes for active geo-redundancy. This ensures business continuity during single-node failures and provides users with nearby access for better performance.

CEN cross-region connection with fine-grained QoS traffic control

Code flow:

1. Create two VPCs on the cloud, one in the China (Hangzhou) region and the other in the China (Beijing) region.

2. Create a TR in Hangzhou and a TR in Beijing, and connect them to the corresponding VPCs.

3. Create a cross-region connection between the Hangzhou TR and the Beijing TR to enable network communication between the two VPCs.

4. Create a traffic marking policy and a QoS policy to set bandwidth limits for different traffic types based on marks. This ensures sufficient bandwidth for various services.

The following resources are created:

• Two VPCs, each with three vSwitches

• Two TRs

• One traffic marking policy

• One QoS policy

Build a cross-region network between an on-premises IDC and a remote VPC using a TR

Code flow:

1. Create two VPCs, one in the Hangzhou region and the other in the Beijing region.

2. Create a TR in Hangzhou and a TR in Beijing, and connect them to the corresponding VPCs.

3. Create two VBRs. Each VBR connects the Hangzhou IDC to the Hangzhou TR.

4. Create a cross-region connection between the Hangzhou TR and the Beijing TR to build the cross-region network.

The following resources are created:

• Two VPCs, each with two vSwitches

• Two VBRs

• Two TRs