Cloud Network Well-architected Design Guidelines:Use Express Connect circuits to build a hybrid-cloud or multi-cloud network

Summary

This topic describes how to use Express Connect circuits and Alibaba Cloud networking services to establish communication between cloud networks and on-premises networks and between multiple clouds. The solutions help you build a hybrid-cloud or multi-cloud network that supports high security, stability, and scalability.

This topic is intended for technical engineers, including chief technology officers (CTOs), architects, developers, and operations engineers. This topic introduces solutions to building hybrid-cloud or multi-cloud networks. You can adjust the network design based on your business requirements. 

Keywords

• Express Connect circuit: Express Connect circuits are cables or optical fibers that connect data centers. Express Connect circuits are typically deployed and maintained by Internet service providers (ISPs). Express Connect circuits are classified into dedicated Express Connect circuits and shared Express Connect circuits based on the deployment mode.

  • Dedicated Express Connect circuits: Enterprises can use dedicated Express Connect circuits to connect on-premises data centers to Alibaba Cloud access points. This solution ensures that your Express Connect circuit is exclusive to you. You can apply for Express Connect circuits in the Express Connect console. This solution is ideal for medium-scale and large-scale enterprises that require high bandwidth, high security, and high reliability.

  • Shared Express Connect circuits: The access points of Express Connect partners are already connected to the access points of Alibaba Cloud. You only need to contact an Express Connect partner, which can deploy an Express Connect circuit between your data center and the data center of their access points. In this solution, multiple tenants share the Express Connect circuit between the partner and Alibaba Cloud. This solution is ideal for medium-scale and small-scale enterprises that do not require high bandwidth.

• Express Connect: Express Connect is a networking service that connects data centers to Alibaba Cloud. You can use Express Connect to establish high-speed, reliable, and secure private connections between data centers and cloud networks. Express Connect improves quality and security for network communication because data transmission over Express Connect circuits is reliable and controllable.

• Virtual border routers (VBRs): VBRs are an abstraction of Express Connect circuits that are isolated and virtualized by using the Layer 3 overlay and vSwitch technologies in the Software Defined Network (SDN) architecture. VBRs are deployed between customer-premises equipment (CPE) and VPCs to forward data from VPCs to data centers.

• Express Connect Router (ECR): An ECR is an important service component that forwards network traffic in a global hybrid cloud in which networks are connected over Express Connect circuits. An ECR provides features such as dynamic routing-based networking and centralized management for route advertisements. For example, you can associate VBRs with an ECR and then associate the ECR with transit routers to allow your data center to communicate with cloud resources.

• Cloud Enterprise Network (CEN): CEN is a high availability network built on Alibaba Cloud global private networks. CEN uses transit routers to establish inter-region connections between VPCs or between VPCs and data centers. CEN helps you create flexible, stable, and enterprise-class networks in the cloud.

• VPC: VPCs are private clouds that customers create on Alibaba Cloud. VPCs are logically isolated from each other. You can create and manage cloud services, such as Elastic Compute Service (ECS), Server Load Balancer (SLB), and ApsaraDB RDS, in your own VPC.

Design highlights of the architecture:

• Stability: The Express Connect circuits forward only internal traffic of the enterprise. Stability is business-critical to the hybrid-cloud or multi-cloud network connections. If the connections fail, communication between cloud resources and between cloud resources and on-premises resources also fails. In this case, your business may experience adverse impacts because business-critical services may become unavailable. Therefore, stability is the key to designing a reliable multi-cloud or hybrid-cloud network architecture.

• Scalability: The bandwidth required by an enterprise varies based on the business scale or cloud adoption stage. When Alibaba Cloud designs a network architecture, Alibaba Cloud takes into consideration the scaling requirements of the enterprise to help the enterprise smoothly migrate to the cloud and increase resource utilization. This design supports on-demand resource scaling, which is more cost effective.

• Security: A multi-cloud or hybrid-cloud network requires network communication between different network domains. In addition, different businesses within an enterprise have different security requirements. For example, access to business-critical services is granted based on the principle of least privilege (PoLP) to ensure data security. To prevent data leaks and abuse, enterprises have security requirements for their internal data.

Stability

Stable bandwidth of Express Connect circuits

Express Connect circuits support bandwidth that ranges from 50 Mbit/s to 100 Gbit/s. If you require higher bandwidth, you can enable link aggregation to increase the bandwidth capacity. We recommend that you select an Express Connect circuit that can provide sufficient bandwidth to ensure service continuity after failover. Meanwhile, you can configure alert rules in CloudMonitor to manage Express Connect quotas. For example, you can use CloudMonitor to prevent packet loss caused by bandwidth exhaustion. For more information, see Configure an alert rule.

Reliable connections over Express Connect circuits

In some scenarios, you may need to temporarily close some connections in a multi-cloud or hybrid-cloud network for reasons such as network maintenance. Before you close connections, we recommend that you choose a high reliability mode based on your business requirements. For more information, see High reliability mode.

To ensure stability, we recommend that you apply for two Express Connect circuits and connect to two access points when you build a multi-cloud or hybrid-cloud network. Meanwhile, we recommend that you enable BGP dynamic routing, instead of static routing, between the VBR and data center or between clouds. Dynamic routing supports automatic switchover in cases of circuit failures. When multiple connections are established, Alibaba Cloud provides multiple redundancy solutions to ensure service high availability:

• Use BGP dynamic routing and bidirectional forwarding detection (BFD) to establish active/standby connections. For more information, see Establish active/standby connections between a data center and Alibaba Cloud by using an ECR.

• Use BGP dynamic routing and BFD to establish active/active connections. For more information, see Establish active/active connections between a data center and Alibaba Cloud by using an ECR.

Other than high availability solutions based on Express Connect circuits, Alibaba Cloud allows you to combine Express Connect circuits with IPsec-VPN connections. For more information, see Configure active/standby connections by using IPsec-VPN (transit router associated) and an Express Connect circuit. If you have a limited budget, we recommend that you use IPsec-VPN connections as standby connections for Express Connect circuits. For the limited capacity of IPsec tunnels, we recommend that you configure IPsec-VPN connections for business-critical services. In addition, we recommend that you enable BGP dynamic routing for the IPsec tunnel to monitor the availability of the IPsec tunnel and automatically converge routes.

Failure drills

Express Connect performs fault drills to simulate faults. For example, Express Connect simulates a faulty connection over an Express Connect circuit to test whether network traffic can automatically switch to the other connection. You can perform fault drills to test the reliability of hybrid-cloud networks between data centers and Alibaba Cloud. For more information, see Use the failure drill feature.

Performance and stability

Express Connect circuit specifications

• Dedicated Express Connect circuits: 1 Gbit/s and lower, 10 Gbit/s, 40 Gbit/s, and 100 Gbit/s.

• Shared Express Connect circuits: 50 Mbit/s, 100 Mbit/s, 200 Mbit/s, 300 Mbit/s, 400 Mbit/s, 500 Mbit/s, 1 Gbit/s, 2 Gbit/s, 5 Gbit/s, 8 Gbit/s, 10 Gbit/s, 20 Gbit/s, 40 Gbit/s, 50 Gbit/s, 60 Gbit/s, 80 Gbit/s, and 100 Gbit/s.

  Note

  • The highest bandwidth that you can apply for in the Express Connect console is 10 Gbit/s. If you require higher bandwidth, submit a ticket.

  • The highest bandwidth for a shared Express Connect circuit that you can apply for in the Express Connect console is 1 Gbit/s. If you require a higher bandwidth, submit a ticket.

Express Connect circuit scale-ups

• Scale up dedicated Express Connect circuits: If the bandwidth required by your business requires is higher than the Express Connect circuit specification, you can converge different Express Connect circuits by using equal-cost multipath (ECMP). ECMP connects multiple Express Connect circuits to the same access point device and associates the Express Connect circuits with the same VBR to increase the bandwidth capacity. For more information, see Configure ECMP between an on-premises data center and Alibaba Cloud. 

• Scale up shared Express Connect circuits: You can change the specification of a shared Express Connect circuit based on your business requirements. For more information, see the Modify the bandwidth limit of a hosted connection section in the Modify bandwidth limits topic. 

Bandwidth limits between ECRs and transit routers

The maximum bandwidth between an ECR and a transit router in the China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), and Singapore is 50 Mbit/s. The maximum bandwidth in other regions is 10 Gbit/s. If you require higher bandwidth, contact your account manager.

Latency

A single region provides multiple access points in different locations. The network latency between an access point and a zone in the same region is less than 5 milliseconds. If your business requires lower network latency, submit a ticket to consult the nearest access point to your ECS instance.

Failover performance

Express Connect performs failovers by using multiple methods, such as BGP dynamic routing, BFD, and failover groups.

• BGP dynamic routing uses automatic convergence to perform connection failovers within seconds.

• BGP dynamic routing combined with BFD can detect connection failures within milliseconds and perform connection failovers within seconds.

• BGP dynamic routing combined with BFD and failover groups can perform connection failovers within milliseconds.

We recommend that you enable BFD and failover groups for BGP dynamic routing to accelerate route convergence and failovers.

Security

Multi-layer protection

Network security is critical to a multi-cloud or hybrid-cloud network. You can use security groups, access control lists (ACLs), and multiple route tables of a transit router to enable multiple types of protection for your network.

• Security group: A security group is a virtual firewall that controls inbound and outbound traffic for ECS instances. You can configure inbound rules for a security group to control traffic to ECS instances in the security group and outbound rules to control traffic from the instances. For more information, see Security groups for different use cases.

• Network ACLs: Network ACLs allow you to regulate access control for VPCs. You can create network ACL rules and associate a network ACL with a vSwitch to control inbound and outbound traffic of ECS instances in the vSwitch. You can restrict access from some data centers to a VPC. For more information, see Manage communication between a data center and a VPC.

• Multiple route tables of a transit router: You can configure multiple route tables for a transit router to connect and isolate data centers and VPCs and control traffic routing between data centers and VPCs. This enhances security for communication between cloud resources and on-premises resources. For more information, see Enable secure network communication using Enterprise Edition transit routers.

• Multiple route tables of a transit router + Cloud Firewall: You can configure multiple route tables for a transit router to isolate trusted network traffic from untrusted network traffic and use Cloud Firewall to detect and filter untrusted network traffic.

Data encryption:

After you establish private communication between a data center and a VPC by using Express Connect circuits, data transmission over the connections is not encrypted. If your business requires higher security, you can create IPsec-VPN connections to encrypt data transmission. For more information, see Encrypt private connections by using BGP routing.

Observability

Observability is classified into the following aspects based on scenarios:

• Observability on connectivity of Express Connect circuits: Configure alert rules in CloudMonitor to monitor the status of connections over Express Connect circuits. CloudMonitor shows whether the connections are available or unavailable. For more information, see Monitoring and alerting for connections over Express Connect circuits.

• Observability on data usage of Express Connect circuits: View outbound data transfers during different time ranges in the Express Connect console. For more information, see Outbound data transfer fees.

• Observability on bandwidth usage of Express Connect circuits: Express Connect is interfaced with CloudMonitor to monitor the status of VBRs and inbound and outbound data transfer rates in real time. You can also configure alert rules so that you are notified of anomalies at the earliest opportunity. You can evaluate the bandwidth usage of an entire Express Connect circuit based on the sum of inbound and outbound data transfer rates of VBRs at the same point in time. For more information, see Monitoring and alerting for VBRs.

• Observability on top N data transfers: Network Intelligence Service (NIS) supports analysis on hybrid cloud traffic. If you use CEN to build a network, you can use NIS to analyze and display the inbound and outbound data volumes from multiple dimensions, such as IP addresses, ports, and protocols. You can also view the top N ports, top N peer ports, and top N protocols that generate the largest volumes of data transfers in a specified region within a specified time range. For more information, see Work with the hybrid cloud traffic analysis capability.

Custom services

Quick detection of anomalies

• You can configure alert rules and specify thresholds for Express Connect circuits and VBRs in CloudMonitor to detect anomalies at the earliest opportunity. For more information, see Configure an alert rule. 

• You can subscribe to NIS, which provides the event center to help you monitor resources based on events. You can view the resources that are exposed to potential risks and configure alert rules for specific events. This way, you can handle these events at your earliest opportunity to prevent business interruptions. For more information, see the Express Connect section in the Manage events topic.

• By default, NIS performs network diagnostics on the entire architecture. Diagnostic items include stability, security, performance, cost optimization, and operational excellence. The inspection results show whether the multi-cloud or hybrid-cloud network is exposed to risks. For more information, see Network inspection.

Detection for inaccessible hybrid-cloud or multi-cloud networks

If your hybrid-cloud or multi-cloud network becomes inaccessible, use the following methods to troubleshoot the issues:

• Method 1: Use NIS to perform path analysis and identify the cause of error. After you perform path analysis, NIS automatically generates the detailed information about virtual paths between VPCs and data centers. If the destination is inaccessible, the system locates the error, checks for the causes, and displays the traffic path between the source and destination. If the issue persists, submit a ticket.

• Method 2: On the gateway devices in your data center, perform connectivity tests between the customer-side IP address and Alibaba Cloud-side IP address of the Express Connect circuit. Run the ping command to test whether the IP addresses can directly access each other. If not, report the error to your Internet service provider (ISP). You can also submit a ticket to report this error to Alibaba Cloud. Alibaba Cloud checks for errors on the Express Connect circuit. User operations separate from Alibaba Cloud operations on the port of the Alibaba Cloud-side switch. If the access device runs as expected but the issue persists, contact your ISP for troubleshooting.

Establish communication for critical businesses in a multi-cloud or hybrid-cloud network

Core architecture:

• Dual Express Connect circuits and dual access points: Apply for resources in two access points and two Express Connect circuits. The Express Connect circuits can implement ECMP load balancing and function as active and standby circuits to ensure reliability and performance.

• An ECR designed based on dynamic routing and a distributed architecture: The ECR can efficiently improve route configuration and management, reduce network latency between Express Connect circuits and zones, and increase the bandwidth capacity between the transit router and ECR.

• The transit router can isolate the ECR from VPCs and connect the ECR to VPCs on demand.

• Data centers connect to third-party clouds and Alibaba Cloud by using BGP dynamic routing and BFD.

  

Establish communication for noncritical businesses in a multi-cloud or hybrid-cloud network

Core architecture:

• Express Connect circuit + IPsec-VPN connection: The active connections is created on the Express Connect circuit and the IPsec-VPN connection functions as the standby connection. If the Express Connect circuit fails, traffic is automatically switched to the IPsec-VPN connection to maintain service continuity. This also reduces connection costs for multi-cloud or hybrid-cloud networks.

• An ECR designed based on dynamic routing and a distributed architecture: The ECR can improve route configuration and management, reduce network latency between Express Connect circuits and zones, and increase the bandwidth capacity between the transit router and ECR.

• The transit router can isolate the ECR from VPCs and connect the ECR to VPCs on demand.

• Data centers connect to third-party clouds and Alibaba Cloud by using BGP dynamic routing and BFD.

  

Establish communication for critical businesses in a multi-cloud or hybrid-cloud network

Coding process:

1. Dual Express Connect circuits and dual access points: Apply for resources in two access points and two Express Connect circuits. The Express Connect circuits can implement ECMP load balancing and function as active and standby circuits to ensure reliability and performance.

2. An ECR designed based on dynamic routing and a distributed architecture: The ECR can improve route configuration and management, reduce network latency between Express Connect circuits and zones, and increase the bandwidth capacity between the transit router and ECR.

3. The transit router can isolate the ECR from VPCs and connect the ECR to VPCs on demand.

4. Data centers connect to third-party clouds and Alibaba Cloud by using BGP dynamic routing and BFD.

Required resources:

• Two VPCs

• Four switches

• One CEN instance

• One transit router

• One ECR

• Two VBRs