By Chen Jianyong, Senior Technical Expert at Alibaba Cloud
This article introduces the R&D background and application scenarios of Smart Access Gateway for Alibaba Cloud network products, the reasons for using Vector Packet Processing (VPP) as the technology foundation based on the project, and which aspects we have made optimization and choices in.
The idea of the project was to realize a product similar to a VPN network gateway. The product is used in three typical application scenarios: hybrid cloud, retail chains, and the interconnection between a headquarters and its branches. The goal was to implement the migration to Alibaba Cloud and interconnection upon power-on.
Based on these product requirements, the technical product needed to support Internet Key Exchange (IKE), route-based VPNs, IP forwarding, and VRF as well as have a high forwarding processing capacity. We eventually chose VPP as the technical base because our surveys found that it matched our requirements at a relatively high level.
However, VPP is just a framework and POC, and is far away from being a real product. We will show in which aspects we have made optimization later.
We chose the VPP baseline version 18.01, which included some basic fragmentation functions. However, these fragmentation functions are mainly used on the ip4-map node, and the fragmentation feature does not support basic forwarding. As a result, we add the ip4-frag node after the ip4-rewrite node to support fragmentation in common forwarding processes.
No code inside VPP 18.01 supports fragment assembly, but a patch outside VPP 18.01 does. Therefore, we integrated this patch into VPP 18.01 to enable such support. This patch is currently combined into the 18.04 branch in the community.
For IPsec fragmentation, fragmentation before encryption is different from fragmentation after encryption. In general, we can choose fragmentation before encryption so that the decryption end can directly send fragments out after decryption to let the client do the assembly work. However, we always need to perform a packet back-to-back IPsec test when developing devices. In this case, the tester cannot recognize the messages that the receiving port receives because these messages are fragmented. To get the testing performance data, the decryption end needs to send out the same messages as the sending end. In general, the encryption end first encrypts and then fragments 1,500 bytes of received messages. Therefore, the decryption end can assemble fragments, decrypt the received messages, and send out the same 1,500-byte messages as the sending end. We support both scenarios.
Because we are working on a commercial product, in addition to forwarding messages, the device needs to support the VPN IKE negotiation, the DHCP server feature, SSH or Telnet management, and some other features. At this point, all messages are sent to applications on devices. However, because Ethernet ports on devices have been taken over by the Data Plane Development Kit (DPDK) under VPP, the received messages need to be differentiated from the self-messages and then sent to apps. At the same time, messages sent by apps need to be read into VPP and sent out through Ethernet ports.
The implementation methods include KNIs, tap interfaces, and AF_PACKET. We chose to take the tap-inject module out of the patch of router-plugin in VPP, and implement the reception and delivery of self-messages by using tap-inject. Specific nodes are as follows:
Of course, we have encountered some problems. For example, reading and writing tap interface data are done through system calls, which has some influence on the message processing of VPP. Another problem is that we do not have too many messages. We will consider porting KNIs to make improvements.
As we previously mentioned, a lot of services are running on our devices. As a result, information synchronization is required among these apps. Let's consider a typical example. The agent receives data from the controller and sends received data to IKED for negotiation. After negotiating keys, IKED needs to send these keys to VPP for decryption. This requires an inter-process communication mechanism.
There are several options, such as simple UNIX sockets and NetLink sockets, and the complex TIPC protocol. We decided to use nanomsg for inter-process communication, because nanomsg provides high supportability for synchronizing a variety of information types due to its simplicity and support for communication patterns such as req/response and pub/sub.
VPP has already supported IKE, but only IKEv2. Even IKEv2, it can be only used as a responder instead of a request side. Additionally, VPP is not suitable for integrating complex IKE negotiations. Based on the preceding situation and considerations, we have prepared a separate IKE app on device providers' side for negotiation. This app then works with VPP to perform the synchronization of security associations (SAs).
There are several IKED options, including strongswan, freeswan, and ipsec-tools.
In scenarios like chains, some customers require support for 4G. In general, drivers of the 4G module receive and send packets based on kernels. Two common ways are used to allow VPP to support 4G. One is based on AF_PACKET and uses AF_PACKET as the messaging "bridge" between VPP and Linux kernels. This method requires small changes, but has poor performance. The other method is to modify the drivers of the 4G module to support DPDK. This method has high performance because it allows direct arrival at VPP without going through kernels. However, this method has a disadvantage, that is, drivers need to be significantly changed and changes are required each time a 4G module is replaced.
You can choose a proper technical solution based on your own requirements.
In general, VPP is a good framework that enables high performance and many features. However, it is only suitable for POCs, and has many disadvantages. For commercial use, you need to know VPP well, be familiar with it, and get ready to overcome its disadvantages.
The LinuxCon + ContainerCon + CloudOpen China 2018 (LC3) open source summit hosted by the Linux Foundation, a world-renowned open source community, opened in Beijing on June 25, 2018. This year, Alibaba Cloud attended this conference as a platinum partner. Three technical experts from the Alibaba Cloud network team shared their experience in their field of expertise including the optimization of open-source platforms and the application of network big data.
High-Performance DPDK-based Server Load Balancing for Alibaba Singles' Day Shopping Festival
19 posts | 11 followers
Follow5544031433091282 - October 3, 2023
Alibaba Clouder - January 4, 2017
Alibaba Clouder - July 23, 2019
Alibaba Container Service - August 25, 2020
Alibaba Clouder - September 6, 2018
Alibaba Clouder - August 10, 2020
19 posts | 11 followers
Follow
Accelerated Global Networking Solution for Distance Learning
Alibaba Cloud offers an accelerated global networking solution that makes distance learning just the same as in-class teaching.
Learn More
Realtime Compute
Realtime Compute offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn More
ApsaraVideo Media Processing
Transcode multimedia data into media files in various resolutions, bitrates, and formats that are suitable for playback on PCs, TVs, and mobile devices.
Learn More
Networking Overview
Connect your business globally with our stable network anytime anywhere.
Learn MoreMore Posts by AlibabaCloud_Network
