Application practice of Coolbpf

1、 EBPF CORE and development model comparison

With the development of BPF technology, developing a BPF program has become more and more simple. Although BPF has improved its convenience, BPF has also been pursuing another aspect: portability. BPF portability is defined as a BPF program successfully written and verified by the kernel, which can run in different kernel versions.

There are two challenges in migrating BPF:

1. The memory layout of different kernel versions is different.

2. The kernel type and data structure are constantly changing, and the structure field may be removed or renamed.

BPF CO-RE (Compile Once - Run Everywhere) is a means to achieve portability.

To support CORE, we provide the following components:

① BTF: describes the kernel image, and obtains the key information of the kernel and BPF program type and code

② Clang releases the bpf program relocation information to the. btf section

③ Libbpf CO-RE relocates the bpf program according to the. btf segment

There are three types of information that need to be relocated:

① Structure related relocation is closely related to BTF. Clang passed__ builtin_ preserve_ access_ Index() records the member offset.

② The variable relocation of map fd, global variables (data, bss, rodata), and extern mainly depends on the relocation mechanism of ELF to update the imm field of the eBPF instruction.

③ The sub-function relocation is to put the sub-functions called by the eBPF program together with the main function, so that they can be loaded into the kernel together.

The development steps of BPF CORE using libbpf are as follows:

Step 1: generate the header file vmlinux. h with all kernel types, and generate it through bpftool.

Step 2: use Clang (version 10 or later) to compile the source code of BPF program into. o object file;

Step 3: generate the BPF skeleton header file bpftool gen command from the compiled BPF object file;

Step 4: Include the generated BPF skeleton header file in the user space code;

Step 5: Compile user space code, then embed BPF object code without publishing separate files.

There are roughly the following function calls:

① __ Open (): Create and open the BPF application, and then set the ski ->rodata variable.

② __ Load (): Initialize, load and verify the BPF application part.

③ __ Attach(): Attach all BPF programs that can be attached automatically. When there are events and network operation messages, the bpf program will be triggered to run.

④ __ Destroy (): Detach all BPF programs and use all resources they use.

There are three main development methods of eBPF, each with its advantages and disadvantages.

1. The kernel comes with sample code: based on the kernel samples/bpf sample code, without CORE. This method does not have any open source projects based on third parties, and the resource consumption is low. However, the disadvantage is that you need to completely rebuild the project yourself, which is inefficient and has poor version compatibility.

2. BPF CORE: based on bpf written by libbpf_ core_ Read code, the development machine generates the binary program corresponding to the target machine. This method does not rely on the deployment of Clang/LLVM in the environment, and takes up less resources. However, a compilation project needs to be built. Some codes are relatively fixed and cannot be dynamically configured.

3. BCC: Based on the most widely used open source project, the development efficiency is high. However, every time you run, you need to execute the Clang/LLVM compilation, and there is a scramble for memory, CPU and other resources; The target environment depends on the corresponding kernel header file.

2、 Functions and architecture of Coolbpf

Coolbpf provides remote compilation (cloud compilation). Remote compilation means that the target machine running the program is not on the same machine as the compiler, which can solve the problem of resource occupation; Local compilation and basic library encapsulation are provided to facilitate users to call the basic library for writing; Support for lower version kernel is provided; It provides automatic generation and release of BTF, and users can use it directly without manual adaptation; It provides automated testing and supports application development in high-level languages such as Python/Go/Rust.

Coolbpf naturally supports the CORE capability of BPF, which solves the problems of compilation and resource consumption. At the same time, the complex libbpf development steps described above have been completely simplified. Users only need to focus on their own function development, not on environment construction and redundant code development.

Coolbpf provides a standardized bpf compilation service. First, when submitting bpf. c to the remote compilation server, the server will return the point bpf. so or bpf. o for different languages according to the kernel version to provide services for advanced applications. Therefore, in your high-level language code, you only need to load bpf.so to run the program, instead of manually triggering the open(), load(), attach() and other functions of Libbpf, it is automatically completed by init() of the high-level language program, so that users can quickly build and deploy the program, just focus on the processing after data output.

Coolbpf also supports running on low kernel versions without eBPF features, and helps it run safely on low kernel versions through the eBPF driver we provide. We implement the verification part of eBPF verifier on the higher version in a driver, which will perform various security checks to ensure the security of eBPF compared with the kernel module. In addition, we convert the original libbpf-based calls to IOCTL system calls.

The previously supported helper functions, creating maps, and loading programs will all be converted into kprobe or tracepoint implementations on lower versions. In addition, perf event and jit are also supported. This allows the same user program to load such a driver and run safely on the lower version kernel without modifying the eBPF program code.

3、 Coolbpf's Network Application Practice

Raptor is a system observable tool based on Coolbpf, which can run on lower versions of kernels such as alios, CentOS 3.10, etc. It can be used as an SDK for third parties to collect data.

In this network application observation, the interactive data content and five-tuple information are determined by monitoring the data interaction, request and reply information in the system call, and sent to the user state through the interactive mode of map, so as to achieve non-intrusive observation, and finally present the observation results such as traffic statistics, request delay, etc.

Let's look at a specific problem to understand how Coolbpf finds the network jitter problem in the packet receiving phase. We know that network packet collection is divided into two stages:

Stage 1: The OS sends the packet to the application's packet receiving queue through soft interrupt and notifies the process to complete the packet receiving of the protocol stack.

Stage 2: The application obtains data from the packet receiving queue after being notified.

We write a BPF program in Coolbpf, and only need to monitor two tracepoints: tcp_ probe，tcp_ rcv_ space_ Adjust can query the delay problem in phase 2.

In this case, a service application received packets slowly. It was found that the kernel side had received the tcp packet, but the application side received it nearly 1 second later. Observation method: deploy the eBPF agent and find that the "packet receiving delay time" in phase 2 is nearly 1 second.

Determine the reason: each time the delay occurs, it is about 42 seconds at a certain time. It is suspected that the delay caused by the application itself is related to a certain scheduled task of the business. Finally, it is found that a certain task of the business will collect the parameters of the jvm on a regular basis, and there is a stop operation for the business.

Solution: After stopping the task, the jitter problem is eliminated.