Alibaba IPv6 large-scale deployment practice

01 Preface

Careful people may have found that when we first opened Taobao, Tmall, Alipay and domestic mainstream apps, we could see the sign of "IPv6", which means that China's mainstream apps have officially entered the era of dual stack of IPv4 and IPv6. This article will share with you our experience in this process from the perspective of APP and cloud products to provide reference for further promoting large-scale IPv6 deployment. Before officially introducing the experience, let's briefly introduce some basic information about IPv6.

1.1 What is IPv6?

IPv6 is the abbreviation of Internet Protocol Version 6, where Internet Protocol is translated as "Internet Protocol". IPv6 is the next generation IP protocol designed by the IETF (Internet Engineering Task Force) to replace the current version of IP protocol (IPv4). IPv6 extends the length of 32-bit addresses in IPv4 to 128 bits. Using IPv6, every grain of sand in the world can be assigned to an IP address.

The original intention of IPv6 is to solve the problem of IPv4 address exhaustion, and make a lot of improvements to IPv4, and finally replace IPv4. However, due to the extensive application of NAT and other technologies, IPv4 has long been the main part of Internet traffic, and the use of IPv6 has grown slowly.

1.2 National strategic value of IPv6

IPv6 is an inevitable trend of Internet upgrading and evolution, an important direction of network technology innovation, and a basic support for building a network power. At present, IPv6 network construction and application innovation continue to accelerate worldwide. Major developed countries and regions have issued IPv6 development plans and policy guidance. Large network operators and application service providers are also carrying out large-scale commercial deployment of IPv6. At the same time, with the gradual maturity of 5G and IoT technologies, the era of the Internet of Everything is coming. Apple's AppStore audit requires full support for IPv6, which makes the market's call for IPv6 rising.

IPv6 is of great significance for China's digitalization and supporting the construction of a network power. In this regard, Academician Wu Hequan, Director of the Expert Committee on Promoting IPv6 Scale Deployment in China, has fully discussed this.

1.3 Social Value of IPv6

In everyone's impression, IPv6 may still be in the early stage of deployment, and major operators are still in the process of network transformation. It seems that it is far away from individuals to fully popularize IPv6. However, according to the data of the national IPv6 development monitoring platform, the total number of active IPv6 Internet users in China has reached 683 million, and the proportion of active IPv6 Internet users in China has reached 66.18%. IPv6 has been unconsciously integrated into everyone's life. With the large-scale deployment of IPv6, what changes will it bring to us?

With the arrival of the 5G era and the rapid development of emerging fields such as the Internet of Things, industrial Internet, cloud computing, artificial intelligence and big data, the demand for IP addresses of mobile Internet is increasing, and IPv4 can no longer meet the current network environment. IPv6 and 5G complement each other. Their goal is to interconnect as many devices as possible to achieve the ultimate state of interconnection of all things. Combining the technical attributes of IPv6 massive public address and the core advantages of 5G high-speed low latency network, it provides quality assurance for the Internet of Everything.

Ordinary users will gradually enter the era of comprehensively saying goodbye to the intranet. At present, most operators have used NAT technology, which makes users experience instability in remote monitoring, game online, P2P video, device public network interconnection and other aspects, and also restricts the further development of these technologies. IPv6 has endless public IP addresses, so as to realize the real interconnection of everything. In the future, Youku videos we watch may be shared by our neighbors. The nail video conferencing we use will also become faster and clearer because of IPv6. Our home monitoring will also become smooth because of IPv6 support. All this will be gradually realized after IPv6 is fully popularized.

1.4 Difficulties and challenges encountered in the large-scale deployment of IPv6?

✪ 1.4.1 Management challenges of super large projects

• IPv6 transformation involves operators' line and address application, network/computing/storage and other IT infrastructure construction or upgrading, application code logic change and other matters. Upgrading is difficult and complex. How to ensure business continuity? It is necessary to meet the core demands of smooth business upgrading, reducing business interruption time, and even avoiding business interruption.

• The state promotes the acceleration of IPv6 scale deployment. Enterprise website systems, Internet APPs, etc. need to complete IPv6 transformation in a short time. How to quickly complete the upgrade under the premise of ensuring business stability is a challenge that must be faced.

• The application software needs to be transformed in a large scale, which will also bring operators' line costs, IT infrastructure software/hardware purchase or upgrade costs, integration service costs and other comprehensive costs. How to improve the transformation efficiency, while achieving sustainable cost controllable and predictable is also a key point.

✪ 1.4.2 Technical difficulties in the new scenario

• From IPv4 single stack to IPv4 and IPv6 dual stack environments, the work corresponding to the underlying image will be doubled. How to stably and efficiently support business development is the first problem to be solved.

• The dual stack network environment increases the complexity of testing. How to establish a convenient testing tool to facilitate daily testing and find exceptions in time.

• As the dual stack deployment terminal is facing a more complex environment, how to ensure user experience in a complex network environment is a technical problem that must be solved in the large-scale deployment process.

✪ 1.4.3 Establishment of normalized operation system

• Large scale deployment is not the ultimate goal. IPv6 needs to move from being usable to being usable, and it needs to continue to operate. The iterative updating of Internet technology is very fast. The version will be released iteratively once to twice a week. How to ensure that the subsequent iterative upgrading will not affect the process of large-scale deployment? This part of the work needs to be integrated into the daily normalized operation system.

1.5 Alibaba's overall thinking and transformation plan for IPv6

No matter from the policy orientation, market demand, or focusing on technological evolution, IPv6 will begin to enter the stage of large-scale deployment. It is against this background that Alibaba takes the lead in comprehensively entering the "IPv6 era". The Group established its IPv6 project team in 2017, and mainstream APPs have entered the large-scale deployment of IPv6.

• Start phase I in 2018: application access layer transformation.

• Starting phase II in 2021: IPv6 upgrading pilot for intranet applications, exploring the best practices for IPv6 evolution.

• Launch phase III in 2022: IPv6 Only pilot.

This article will focus on sharing Alibaba's mature application access layer transformation experience, and will gradually share the transformation experience of IPv6 upgrade and IPv6 Only pilot for intranet applications. IPv6 large-scale deployment is not only a super project, but also a technological innovation project.

Next, we will expand in four dimensions:

1. Overall planning;

2. Analyze this super complex project from the four levels of cloud, management and end (server and client);

3. Three core technologies created to overcome the difficulties of new scenes;

4. A normalized operation system for IPv6 normalized operation.

02 Overall transformation scheme

2.1 Strategic planning

The overall development of IPv6 can be summarized into three strategic stages:

The first strategic stage is the initial stage of IPv6 development, which is more of an isolated experimental environment.

The second strategic stage is the coexistence stage of IPv6 and IPv4, which is divided into two stages: IPv4 dominated stage and IPv6 dominated stage. The emphasis of these two stages is slightly different. In the dominant stage of IPv4, we focus on increasing the proportion of IPv6 traffic in the public network. Because the coverage is very large and there is no business value in a short time, we need to have strong policy traction. At the leading stage of IPv6, the enterprise's independent driving force will be enhanced, and end-to-end transformation will be the focus. We are now in the process of the evolution from IPv4 dominated to IPv6 dominated in the second strategic stage. Therefore, it is urgent to promote the large-scale deployment of IPv6, so that the proportion of IPv6 traffic can be greatly improved, and we can enter the IPv6 dominated stage.

The third strategic stage is the maturity stage of IPv6. We need to gradually shut down IPv4 to fully enter the IPv6 Only era.

Complex problems are always daunting. It is a good planning idea to break down problems one by one. Based on the division of these three strategic stages, we can carry out transformation in stages to isolate mutual dependence, which can not only achieve rapid improvement of public network traffic while ensuring the stability and continuity of business, but also carry out innovation and exploration step by step.

1) Strategic promotion: With a long-term strategic plan, the project can be more clearly promoted. Next, we will focus on IPv6 traffic improvement and large-scale deployment. When it comes to the implementation phase, we generally follow the "three-step" strategy. Each phase needs to go through these three steps, which is an important guarantee to ensure the stable operation of our business.

• Step 1: Application pilot, first of all, needs to run through the entire transformation process through one application to get through the dependence of cloud, pipe and end.

• Step 2: Go online on a small scale. In practice, you need to have a refined gray scale strategy, and gradually increase the quantity according to the fineness of provinces, operators and proportions.

• Step 3: It is necessary to form a replicable solution and carry out large-scale promotion and deployment within the group.

2) Organizational guarantee: In order to better implement the overall strategic plan, the Group has set up a strong implementation project team. According to Alibaba's mainstream APP, it has set up 24 subprojects (composed of students from PM, terminal, testing and development). In addition to the subprojects of cloud products (composed of PM, SA and PD development of various products), it has set up an Alibaba IPv6 (cornerstone) project team.

3) Financial support:

• The transformation process will involve hardware transformation and hardware and platform costs. At the same time, the project team is relatively large, and we involve a small amount of operating costs, which require budget planning in advance.

• When hardware equipment and testing mobile phones are involved, relevant equipment should be purchased in advance

2.2 Super large project

Next, we will enter the actual transformation. IPv6 transformation is a systematic project, especially for Alibaba. In order to effectively promote the project, we divide it into three parts from the global perspective: "cloud management end". The transformation of the middle end is divided into the service end and the client end. We separate the two parts, and finally we will elaborate from the four parts: cloud, management, end server, and end client.

© 2.2.1 Cloud product transformation

In order to better improve the public network traffic (the transformation of server dual stack will not be discussed here), we need to fully complete the IPv6 transformation of enterprise Internet domain name servers, support AAAA records and IPv6 domain name resolution requests, and configure all core business domain names with A and AAAA records. Alibaba Cloud HTTP DNS service is used to support the mobile terminal's IPv4/IPv6 dual stack and IPv6 only domain name resolution capabilities.

Cloud product transformation will take better support for APP IPv6 transformation as the primary task, and finally achieve the goal of increasing IPv6 users and public network traffic. It can be divided into three phases. The first phase is IPv6 support at the public network entrance and security management and control place where users access, including DNS, http DNS, CDN, SLB, DDoS, WAF, ACL, IP address library, etc. These products support IPv6, which means that users can provide safe and efficient IPv6 services based on these products. Next, Alibaba Cloud will provide dual stack support on the internal network to achieve end-to-end IPv6 traffic connectivity. Finally, the future evolution goal is IPv6 Only.

⍟ Cloud resolution DNS supports IPv6

To use IPv6 services, DNS must first support IPv6 resolution. As we all know, when users access the Internet, the access request will first arrive at DNS, and DNS will query and return the correct IP address according to the access domain name. In the IPv6 era, Alibaba Cloud still resolves DNS as the entry of cloud computing services, and supports IPv4 and IPv6 dual stack resolution (that is, one domain name resolves two addresses, one IPv4 address, and one IPv6 address), which continues to provide a powerful and stable resolution scheduling entry.

⍟ SLB supports IPv6

The public network entrance will give priority to supporting IPv6, rapidly increasing the proportion of public network traffic. SLB has long been the public network entrance of key business systems, which can distribute traffic to multiple ECSs to improve the availability and processing capacity of business systems.

There are two main products that SLB supports IPv6: First, it adopts an independent IPv6 type SLB. There is no difference in performance and function between independent IPv6 SLB instances and IPv4 SLB instances. Users can quickly create an IPv6 SLB instance by selecting the IPv6 type when purchasing an SLB. Secondly, IPv6 SLB and backend ECS also use IPv4 private network addresses for communication.

⍟ CDN service

CDN Content Delivery Network solves network performance problems across regions and operators, and provides stable and fast acceleration services. CDN is the traffic portal of the Internet. Alibaba Cloud CDN team has started IPv6 transformation since 2017, actively and continuously invested relevant resources in IPv6 transformation, continued polishing in terms of node deployment, node network architecture transformation, IPv6 function support, IPv6 scheduling capability improvement, IPv6 full link monitoring scheme, IPv6 quality evaluation system construction, etc., invested physical resources and human resources, and pushed the overall IPv6 capability closer to IPv4. Up to now, IPv6 transformation and upgrading of nearly 1000 nodes have been completed accumulatively. IPv6 compliance rate exceeds 90%, exceeding the acceptance standard of the Ministry of Industry and Information Technology. In terms of productization, the basic CDN products and the whole station acceleration products are the first to pass the IPv6 Enabled CDN released by the Global IPv6 Forum.

Alibaba Cloud CDN adds IPv6 CDN nodes that have been transformed by operators in various regions to the scheduling domain. Users enable IPv6 functions on Alibaba Cloud console and configure the grayscale scale of image domain names. The total grayscale of IPv6 can be controlled from the traffic portal to ensure that there is no shortage of IPv6 resources or no resource to schedule.

For the 302 scheduling domain name, there will be the problem of IPv6 address abbreviation when jumping. The browser will automatically jump with the abbreviated IPv6 address when accessing. When using libcurl and other network libraries in the APP, the automatic abbreviation will not be triggered, so that the obtained IPv6 address can be requested as it is. The 302 node needs to be configured with two sets of IPv6 VIPs before and after the abbreviation to ensure that it can jump in any scenario. In the HTTPS VIP certificate, IPv6 VIPs need to be signed to ensure the normal jump under https.

For the free flow domain name, the free flow scheduling domain needs to assign IPv6 VIP groups, include all the free flow node IPs that need to be reported to the operator, and have the ability to enable and disable IPv6 functions, to ensure that the IPv6 node is not enabled before the operator completes reporting, and the node can be enabled in time after the operator completes reporting, so as to avoid the failure of free flow or high water level in the free flow scheduling domain.

⍟ WAF service

When users communicate with IPv6, there is also a key cloud service, Web Application Firewall, which is responsible for malicious feature identification and protection of website or APP business traffic and returning normal and safe traffic to the server. When we run the IPv6 protocol, we must upgrade to the protection capability of the dual stack of IPv6/IPv4 to avoid malicious intrusion of the website server, ensure the security of the core data of the business, and solve the problem of abnormal server performance caused by malicious attacks. Alibaba Cloud WAF already supports one click IPv6 upgrade. Under the protected domain name on the WAF console, you can directly turn on the IPv6 status switch.

⍟ DDoS service

Equally important is Alibaba Cloud's DDoS protection service, which is based on Alibaba Cloud's global DDoS protection network and combines Alibaba's self-developed DDoS attack detection and intelligent protection system to provide users with manageable DDoS protection services. It is also necessary to increase the protection capability of IPv6 to automatically and quickly mitigate the impact of network attacks on services such as increased delay, limited access, and business interruption, thereby reducing business losses and potential DDoS attack risks. If you need to perform DDoS protection on IPv6 addresses, you can open the IPv6 protocol in the DDoS protection package, and the protection object is set as the address of the IPv6 conversion service.

⍟ ACL black and white list and security policy

Ensure the integrity and efficiency of IPv6 security system protection, and uniformly upgrade network security equipment such as firewall, intrusion detection, behavior audit, and traffic cleaning to support normal work in IPv6 environment. With the development of IPv6, the number of IPv6 addresses will far exceed that of IPv4, and the capacity of the existing ACL black-and-white list will not be satisfied, so it needs to be expanded and reconstructed in advance. For nodes with IPv6 security protection capability at risk, network security equipment shall be upgraded or replaced. IPv6 security policies are formulated and configured from the application business level and security management level to ensure that IPv6 security policies include all IPv4 policies.

⍟ Other cloud products support IPv6

In addition to the products mentioned above, there are many other products that have been or will be supported, such as OSS for object storage, RDS for database, DTS, API gateway, etc.

© 2.2.2 Pipeline reconstruction

Next, we will enter another key transformation scenario, namely "pipeline transformation". For Alibaba, we need to complete the comprehensive transformation of IDC network and IT network.

Completes the transformation of the group level data center core network and the Internet export IPv6 network.

1) Transformation steps

• First, in order to better support cloud product transformation, the Internet provides IPv6 domain name resolution, load balancing and security services. Core network, security cleaning, MC, BSW and LSW are supported first, followed by the 4to6 address translation of ANAT and XGW, as well as LVS and CDN.

• Then transform the DSW, PSW, ASW, and NC in the IDC cluster to support ECS and Docker services.

• Finally, we started to try IPv6 Only, and entered the IPv6 era in an all-round way.

2) Scheme introduction

• The core network supports dual stack, and there are two main technical directions: physical machine dual stack and MPLS 6PE/6vPE. The physical machine dual stack enables IPv6 and routing protocols on existing IPv4 routers and links. Like IPv4 messages, IPv6 messages are directly encapsulated and forwarded on the link layer; MPLS 6PE/6vPE keeps the existing MPLS network unchanged, and encapsulates IPv6 messages into MPLS for forwarding. This method can change the support on demand in PE equipment.

• The Alibaba core network has enabled MPLS, and the overall evolution direction is comprehensive MPLS. The P node device becomes a BGP free core to significantly reduce the functional requirements for the device and reduce power consumption, which is very helpful for saving Capex and Opex costs.

• Based on the actual situation of Alibaba's core network and the subsequent overall evolution direction, we choose the 6PE/6vPE solution for IPv6 evolution. We upgrade the PE router to support dual stack without changing the existing core network P router. Only PE devices need to enable IPv6, P devices do not need to enable IPv6, and the overall implementation cost is low. As a new service carrier of MPLS, IPv6 has little impact on the existing operation and maintenance. IPv6 maintenance only covers PE, giving full play to the existing efficient operation and maintenance system. In AZ, the dual stack of physical machines is mainly promoted, and the combination of the two technologies can achieve efficient promotion.

Alibaba office network transformation

The transformation of the office network is also a complex process, which not only involves the cost of transformation, but also requires a lot of manpower, so it is a relatively long process. In order to better meet the business needs, we also divide it into two steps.

• Step 1: Two transition plans, first through point deployment, to meet everyone's testing needs. Then, by reforming our Vpn client and gateway, we can obtain IPv6 address by dialing in Vpn to meet the development and testing requirements.

• Step 2: After a year of test preparation, we will gradually start IPv6 transformation in 21 parks across the country in early 2022. It is expected to complete the double stack transformation and upgrading of major parks in China by the end of March 2023.

Deploy the campus network convergence POP node in the national core cities. This convergence point is connected to the core export equipment of each park in the city, and Alibaba Cloud is connected to the corresponding regional access point. It enables the rapid deployment of the resources needed for the campus network on the cloud. With the help of the IPv6 capabilities of the cloud resources, it quickly establishes a new generation of dual stack operation and maintenance system for the campus network, such as DNSv6 services, DHCPv6 services, dual stack fortress machines, etc., eliminating the huge labor and capital costs of offline self construction.

✪ 2.2.3 Application end transformation

We have organized more than 20 APPs of the Group to carry out IPv6 dual stack transformation synchronously. The transformation content will be more focused and the efficiency will be much higher. Let's first introduce the server transformation project.

⍟ New construction, application and transformation

1) Environmental preparation

• Web container environment: choose the latest version of Tengine that supports IPv6 protocol, install toa module, and support transparent transmission of IPv6 header information to application services.

• Development environment and OS system: The operating system supporting IPv6 protocol shall be selected for application development and compilation, such as Windows server above 2003, MAC OS after 10, and CentOS 7 or Alios 7U of Linux system.

• Test environment: The office network environment and the test room are connected through IPv6 dedicated lines. The office network provides IPv6 wireless/wired access points while retaining the original IPv4 access points. Developers can access IPv6 access points for daily development and office transaction processing. If they need to access public network services that do not support IPv6, they can switch to IPv4 network environment. You can also dial IPv6 through Vpn for IPv6 testing.

2) Business application transformation

• Scenario 1 - IP address library: If you use the IP address library service to determine the location of a user's home, you need to upgrade to the latest version of the IP address library data service and have the ability to upgrade regularly. Ensure the accuracy of IPv6 regional attribution judgment.

• Scenario 2 - Uniform IP address format: Because IPv6 addresses can be abbreviated, if they are judged directly by string, IPv6 with and without omission will be judged as not an IP address, resulting in deviation in business processing. At the same time, some browser requests will automatically skim IPv6 addresses, JAVA network packets, CURL, etc., but will not actively skim IPv6 addresses, which will increase the complexity of server processing. Therefore, a public processing is required in advance to standardize all IPv6 addresses through public processing, unify business processing logic, and reduce inconsistency between businesses.

• Scenario 3 - IP address saving: In the general process of logging on the server side and logging on the user information, the user IP address will be saved in the database and other storage. For the database with strict restrictions on data type and length, it needs to be defined according to the storage model. The original IPv4 only needs 32-bit strings or long integer data to be saved, while IPv6 needs to be expanded to 128 bits, and the long integer data cannot be saved, High 64 bit and low 64 bit split storage are required for processing.

• Scenario 4 - Interface transmission: When using the http get method to transmit multiple user IPs in parameter form, you need to pay attention to the upper limit of 1024B of the get. It was originally OK for 10 users to transmit IPv4 together, but now when using IPv6, the IP length of 10 users will exceed the upper limit of the get, and you need to use post or lower the upper limit.

• Scenario 5 - Hardcopy with IP address: The IP address of upstream and downstream call interfaces cannot be written directly in the code or configuration file, because the time for upstream and downstream to complete IPv6 transformation is not the same, and the online time is not the same, which will lead to online failures. All calls need to be changed to domain name mode.

• Scenario 6 - How to obtain the client IP outlet address: In a dual stack environment, the same request can only obtain one IPv4/IPv6 address from the request header, and it is impossible to obtain both. If you want to obtain IPv4 or IPv6 addresses at the same time, you can only choose to repeat requests, or transfer the client address through parameters. The request field needs to be extended, and IPv4/IPv6 is divided into two fields for submission. At the same time, the server needs to do the receiving transformation processing.

• Scenario 7 - Log analysis logic: As we all know, in order to facilitate log analysis and disassembly, all business logs will be defined in a unified format, and the fields of log output will be separated by "| |" or fixed by field length. If the separator is used, it should be considered that it can no longer be used, because IPv6 has this symbol. If the IP address is separated by a fixed length, it is also necessary to consider that the IP address can no longer be fixed to 32 bits. It should be adjusted to 128 bits. At the same time, it should be downward compatible with IPv4, and IPv4 should also be supplemented to 128 bits.

3) Rely on transformation

• Update of the third-party library: Select the third-party SDK version that supports IPv6 protocol. If the third-party library no longer supports IPv6, you need to find a replacement scheme.

• Security deployment environment: application security services such as access layer security controls, current limiting plug-ins, ACL white list plug-ins, etc. shall support IPv6 protocol.

4) Degradation capability construction: It is necessary to consider how to downgrade to IPv4 to continue providing services when IPv6 network is unavailable from the perspective of business logic.