Serverless multilingual application monitoring capacity building

As a basic operation and maintenance capability and a core stability measure, the development and operation and maintenance personnel can effectively locate faults, prevent potential risks, analyze long-term trends for capacity planning and performance tuning through the monitoring system, which is an essential part of the software development life cycle. At the same time, as the best practice and future evolution trend of cloud computing, Serverless is highly respected in the cloud native era because of its fully managed, maintenance-free use experience and pay-as-you-go cost advantage, and will become the core capability provided by cloud manufacturers in the next decade. With the increasing popularity of Serverless and the wider coverage of scenarios, more users who use PHP, Python, C/C++, Node.Js, Golang and other languages begin to upgrade the Serverless architecture.

For these users, the traditional application monitoring scheme has the following pain points:

● High construction cost: it is necessary to deploy a complete set of monitoring systems, including data acquisition, index transmission, persistent storage, visual display, alarm and other modules, which increases additional resource costs and labor costs.

● Strong intrusion of buried points: It is necessary to evaluate the monitoring index demands of various languages, frameworks and interfaces, and introduce three-party dependency to manually bury points. Although for PHP, Python and other languages, there is already a technology to achieve the collection of indicators without modification by means of object (module) replacement enhancement, there is still room for further improvement in terms of capability maturity, framework compatibility, and operational stability.

● Complex operation and maintenance: users need to ensure the low latency, high availability and accuracy of the entire monitoring link, and need to compare and analyze the impact of the introduction of monitoring buried points on the performance of the original application and continue to optimize.

Serverless products need to provide a unified, out-of-the-box, non-intrusive and zero-reform way to achieve application monitoring capabilities in any language, so that multilingual users can fully enjoy the benefits of Serverless technology. Let's first introduce the eBPF (Extended Berkeley Packet Filter) technology behind it.

The full name of eBPF is Extended Berkeley Packet Filter, which started from Linux 3.18 and is a revolutionary Linux kernel technology. EBPF provides the general ability to execute specific code efficiently, safely and non-intrusively based on system or program events. Before the birth of eBPF, due to the isolation between user state and system state, applications could not directly process kernel data. However, if the kernel was modified directly, it would be quite complex. Every development or debugging would require recompilation, which was very inefficient and could not guarantee security.

As a virtual machine running in the kernel, eBPF allows developers to directly submit eBPF programs and run specific functions without modifying the kernel code. Based on the event-driven model, the eBPF program will trigger execution when the kernel runs to a specific hook point. The predefined hook points include system calls, function entry/exit, kernel tracepoints, network events, etc. The non-existent hook points can also be dynamically buried through KProbe and UProbe, providing the tracking ability of kernel and user state functions. With rich hook points, eBPF technology can be widely used in many scenarios, including network monitoring, security filtering and performance analysis.

The workflow of eBPF is shown in the following figure. First, compile the prepared eBPF program into bytecode by using LLVM or GCC in user space, and then load it into the kernel by using the system call bpf. The eBPF virtual machine will use the verifier to verify the security of the bytecode, such as using only limited helper auxiliary functions, limited number of cycles and execution time, and DAG to determine whether there is unreachable code, so as to avoid kernel crash.

After the security check, the eBPF bytecode will be compiled into native machine code through the just-in-time compiler (JIT), providing near-kernel local code execution efficiency, and mounted to the specific hook point. The user status program and the eBPF program communicate in both directions through the memory resident eBPF Map structure. Whenever a specific event occurs, the eBPF program can transfer the collected statistical information to the upper user status application through the Map structure for further data processing and analysis.

SAE (Serverless Application Engine), as the industry's first application-oriented Serverless PaaS platform, is fully managed and maintenance-free, and realizes the Serverless of individual Web applications, microservice applications and scheduled tasks. One of its core advantages is that users can deploy their applications/tasks directly into SAE with low mental burden and zero cost. At present, in the JAVA ecosystem, whether it is code package deployment, the integration of monitoring call chain, or the migration of distributed scheduling framework, users can use it without changing any business logic and version dependency.

Based on eBPF technology, SAE application monitoring supports the non-intrusion indicator collection and monitoring alarm capabilities for any language and framework, provides the golden three indicators RED (request number, error number, response time) and HTTP status code statistics including the application/instance dimension, and provides the interface level call information of services and dependent services. With the help of the built-in visualization platform, the development, operation and maintenance personnel can evaluate the current external service status of the application in a timely manner, and effectively identify problems such as user experience, service interruption, and business exceptions.

SAE application monitoring capability has the following core advantages:

● Free of operation and maintenance: it is used out of the box and takes effect immediately after deployment. Users do not need to operate and maintain the monitoring alarm components.

● Code non-intrusion: rich monitoring data can be obtained without any code burial point and any dependent modification.

● Language independence: network protocol parsing through the kernel layer, supporting any language and any framework.

● Multi-protocol coverage: supports monitoring of indicators and links for HTTP, MySQL, Redis, Kafka, DNS and other network protocols.

● High performance: By reducing the copy of data between the kernel state and the user state, the index data can be obtained with extremely low performance consumption.

In terms of technical implementation, SAE implements multi-tenancy isolation by deploying eBPF probes in the form of sidecar and user business programs in the same security container. The eBPF program will listen to the system calls such as accept/close/read/write/sendto/recvfrom, obtain the local and remote addresses, thread context, file descriptor fd, and other information, and associate the current container instance information to output the original event. Then the content of the original event is sent to the user-state program. The user-mode program identifies the application layer protocol, parses the key fields of the protocol, matches the request and response of a single link, and processes the data through event filtering, dimension convergence, meta information association, pre-aggregation and other steps to generate the final index. Finally, periodically report the collection indicators to Arms Prometheus for persistent storage.

summary

Facing the future, cloud computing will be fully serverless, multilingual, and full ecological support will be the focus of the development of Serverless products. The SAE application monitoring capability will also continue to evolve and enhance. At present, the non-intrusive, multi-dimensional, high-performance application core indicator monitoring and alarm capability has been fully launched. Welcome to use it. At the same time, in the subsequent RoadMap, SAE will introduce advanced functions such as global call topology, upstream and downstream dependency of services, call link details, and create a more complete Serverless multilingual observable system.