eunomia-bpf eBPF lightweight development framework

1、 Summary

There are several pain points in the current development and distribution process of eBPF program:

First, for novices, the threshold for building and developing eBPF programs is high. We must pay attention to the interaction and information processing in both kernel mode and user mode, and also need to write user mode loading code.

Second, various eBPF programs cannot be packaged, distributed and released conveniently and quickly on different kernel versions of different architectures. Many small tools of eBPF are developed in different languages and have different interfaces, which cannot be easily integrated into large observable systems.

Third, there is no good plug-in solution at present. In many cases, the entire observable framework must be recompiled and then redeployed online to update the ebpf probe or data processing module. In addition, if a third-party user-mode data processing code is introduced, the code crash will cause the entire program to crash.

Therefore, we propose two solutions.

First, for beginners, the data exported from the kernel state can be automatically obtained by writing the code in the kernel state, and can be loaded and run after compilation, which reduces the learning cost of eBPF and improves the development efficiency.

Second, based on the feature that libbpf compiles once and runs everywhere, it completely separates the compilation and operation of user-mode and kernel-mode, and distributes them through standard JSON or WASM modules, without recompiling. The application startup takes less resources, takes less time, and even the container startup is shorter.

In addition, we hope to provide standardized distribution methods, such as JSON/WASM distribution. WebAssembly is a binary format based on stack virtual machine, designed for portability. At the beginning, the goal was to enable the code written by C, C++and rust to run in the browser. Up to now, it has developed into a lightweight, high-performance, cross-platform and multilingual sandbox environment, which can be applied to cloud native software or components. Its design idea is similar to eBPF, but the difference is that it runs in kernel mode or user mode.

We try to combine the two. For example, when only writing the kernel code, we can directly package the entire eBPF program into JSON, which contains various configuration information, such as the data memory layout information reported from the kernel state to the user state, such as the global variables to be configured, so that most of the kernel eBPF program configuration information can be completed through JSON only, and can be loaded correctly.

After combining with WASM, the user mode packages the eBPF program through WASM, including data processing, configuration information, complete commands, function parsing, etc. in the user mode, which is distributed with eBPF.

2、 Example

Eunomia is not a complete system, but is similar to the development library and development framework. It can be easily embedded into the coolbpf tool chain, and can also be embedded into other programs as a development library or development framework.

You can download the pre-compiled eBPF program directly from the web page through a one-line command to run. Use WebAssembly or JSON modules for distribution. There is no need to recompile during deployment, and the startup speed is fast.

The form in the figure is put in the URL. It can also be changed into OCI image or Docker image. It can be stored in Docker warehouse or Github package. The use method is basically the same as Docker. You can simply execute pull and run to run, or you can directly use the compiled package.

Compared with the traditional Docker image, its startup speed is faster, and it also retains the important features of eBPF, which can be easily embedded into other programs as sub-modules or plug-ins.

Through Eunomia, you only need to write the kernel-mode code to run correctly, which can minimize the obstacles for novices to get started, omit the user-mode loading framework writing, and automatically export the kernel-mode perf event or ring buffer event. In addition, it is fully compatible with native libbpf. It can obtain the kernel-state code of libbpf tools and run directly without modifying any code.

You can add additional tracepoints or add other content in the form of comments. Using the container to package and compile the tool chain, you don't need to worry about the environment configuration problem. One line of command generates the project template and one line of command compiles.

A complete eBPF tool is usually divided into user space and kernel space. The user space is responsible for loading and reading the statistics and configuration information returned in kernel mode, and can write user mode auxiliary programs in WASM. WASM itself is a safe and efficient user-mode data processing and control logic. When in user-mode, there is no need to worry about interference to other applications, and the speed is very fast.

Users can write in multiple languages, such as C, C++, Go, and Rust, which can be compiled into the form of WASM. We package the kernel-mode code and user-mode code together to generate the WASM module, distribute and manage the eBPF program with the help of WebAssembly's relevant ecosystem, which can be embedded in large applications as programmable modules or plug-ins.

At present, we have implemented the direct compilation of tools written with libbpf in libbpf tools into WASM modules for distribution.

The main difficulty in the process is that the memory layout of WebAssembly, such as the memory layout of the structure, is completely different from that of eBPF, so there may be data serialization or translation overhead in the API. At the same time, because WebAssembly itself runs in the sandbox environment of the virtual machine, the system occupation that can be accessed is also limited, so it must undergo additional conversion.

3、 System architecture

The bottom layer of the architecture relies on the infrastructure in kernel and user mode, such as libbpf library and kernel, which will provide relevant compilation tool chain to help generate JSON or modules packaged into WASM. The tool chain itself uses tools such as Clang/LLVM and bpftool. The dynamic loading library can be used independently, independent of WASM, and can dynamically load the eBPF program according to JSON information. It can be changed into a web service form by putting an HTTP on the outside, realizing the kernel function as your service, and becoming the kernel function and service form.

We also implemented the WASM abstraction layer, including API specifications, such as the access form occupied by the WSAI system of the virtual machine used to extend WASM or the access form interacting with eBPF. There are also libbpf libraries customized based on WASM, portable auxiliary state programs, and serialization libraries, which are used to load libbpf-based eBPF programs in the WASM module.

Runtime libraries can be easily replaced, such as WSI's WASM runtime. In addition, the upper layer also implements LMP, command line tools, observability tools, etc.