Recently, at the Apsara Conference 2022 OpenAnolis Forum-eBPF & Linux Stability Session, Zhejiang University’s Yusheng Zheng (a maintainer of the eBPF Technology Exploration SIG) introduced eunomia-bpf: Lightweight Development Framework for eBPF and Wasm. The following article highlights the main points of that speech.
Hello everyone! I am Yusheng Zheng from Zhejiang University. Today, I would like to introduce the eunomia-bpf project. It is a lightweight eBPF development framework designed to simplify the development, distribution, and operation of eBPF programs, and running eBPF programs in multi languages and Webassembly.
Through some simple examples, I will show how eunomia-bpf can download and run eBPF programs from the cloud with the one-line command, only write kernel code to run and export events, and the combination with WebAssembly (Wasm) and other functions. Finally, I will briefly explain the principles of eunomia-bpf, its design ideas and implementation, and the next development direction.
Eunomia-bpf originated from an idea during the 2022 National College Student Operating System Competition. It aims to run the eBPF program as a service, package the eBPF program as a JSON object, and dynamically plug and unplug any relocatable eBPF program through HTTP requests. It is expected to adapt to different kernel versions and architectures. After the competition, with the help and guidance of several university teachers and some community partners, these ideas were gradually turned into a rudimentary open-source project. (Here, I would like to thank Professor Lijun Chen of Xi’an University of Posts and Telecommunications and the team and Wenan Mao from OpenAnolis.) Currently, the team includes me, the team of Professor Lijun Chen, and friends from OpenAnolis.
eunomia-bpf aims to solve the following two problems (or the main pain points in the development and distribution of eBPF programs):
Therefore, we propose three solutions:
The three parts above are the core features of eunomia-bpf. Let's look at some examples.
Eunomia-bpf is not a complete system but development libraries, compilers and tools that can be easily embedded in a large toolchain (like coolbpf) or embedded as a runtime library anywhere we need to use eBPF as a plug-in.
The precompiled eBPF program can be downloaded directly from the web page url with the one-line command. Use the WebAssembly module or JSON configuration file for distribution the eBPF programs without recompiling during deployment. The startup speed is one to two orders of magnitude faster than the BCC.
In the figure above, the URL used in starting the eBPF program can be replaced by OCI images or Docker images, which can be stored in Docker Repository or GitHub Package. The usage method is virtually the same as Docker. You only need to execute pull and run, or you can directly push the compiled package to use. Compared with traditional Docker images, Wasm, as a lightweight container, has a faster startup speed and retains the important feature of eBPF. It can be easily embedded into other programs as a submodule or plug-in and is completely independent of the specific kernel version and architecture.
With eunomia-bpf, you only need to write kernel mode code to run the program correctly, minimizing the barriers for newcomers to get started. Without writing libbpf user mode loading framework, you can automatically export kernel mode perf event or ring buffer event, for example:
Also, you can automatically collect the data in the hash map and generate a histogram, as shown in the following code:
The corresponding user mode loading code can be automatically generated, and the information in the corresponding hist map can be printed as a histogram by writing the corresponding annotation information in the kernel mode:
You only need to add an annotation to the corresponding global variable position in the code of the kernel mode to automatically generate the command line parameters of the corresponding tool, for example:
The command-line help information is listed below:
In addition, it is fully compatible with native libbpf. After obtaining the kernel mode code of libbpf tools, it can be packaged, released, and run directly without modifying any code.
You can add tracepoints or other content in the form of annotation. You can use a container to package and compile a toolchain without worrying about environment configuration issues. The one-line command generates a project template, and the one-line command does compilation.
Generally speaking, the complete eBPF application is divided into two parts: user-space program and kernel program. User-space program is responsible for loading BPF bytecode to the kernel or reading statistics or event details returned by the kernel to carry out relevant data processing and control.
We can write the user-mode auxiliary program in Wasm to complete safe and efficient user-mode data processing and control logic. It also has the features of eBPF, such as security, portability, lightweight, modularity, etc. (Wasm is also a sandbox environment like eBPF. Even if the Wasm module crashes when it is running in the user mode, it will not cause the crash of the host program.) It can also be used as a plug-in. When adding new data processing logic, you do not need to change the original code. (Note: Wasm is optional rather than necessary. Writing kernel mode code is sufficient for some simple applications.)
We write the code in the C language and package it to generate the Wasm module. After that, we can:
Here is a simple Wasm module that can obtain the signal transmission events between processes of the current system. It can accept some command line parameters and process the reported information.
Currently, we can compile the programs in bcc/libbpf-tools into Wasm modules without code modification. In terms of development experience, it can also be the same as the eBPF program that uses the C/C++/Rust language to develop libbpf. Later, you can also try to introduce other languages to develop SDK (such as Go, Java, etc.).
The main difficulty in combining Wasm and eBPF is that the memory layout of Wasm is different from the eBPF program, and the struct of the C language cannot be mapped directly, so we create some specify toolchain for generating eBPF-Wasm language bindings. At the same time, Wasm has many restrictions on accessing system resources (such as files and networks). Many standard libraries are missing, so we need to carry out some special processing and transplantation in Wasm modules.
The underlying architecture depends on the infrastructure of the kernel mode and the user mode (such as the libbpf library and kernel), providing relevant compilation toolchains to help generate eBPF JSON skeleton or package code into Wasm modules. The toolchains include Clang, LLVM, and bpftool. The dynamic loading library can be used independently and has nothing to do with Wasm. It can dynamically load eBPF programs based on JSON information and can easily implement kernel functions as your service through HTTP interfaces (the form of kernel functions and services).
We have also implemented the Wasm abstraction layer, which contains API specifications (such as the access form occupied by the WSAI system of Wasm) or the access form interacted with eBPF. There are also Wasm-based customized libbpf libraries, ported auxiliary state programs, and serialization libraries, which are used to load libbpf-based eBPF programs in Wasm modules.
Runtime libraries can be easily replaced (such as the Wasm runtime of WSI). In addition, the upper layer implements LMP, command line tools, and observability tools.
The eunomia-bpf project is open-source in OpenAnolis. You are welcome to try it and give suggestions and feedback to improve it.
Please see the following links for more information:
Alibaba Cloud Community - February 3, 2023
Alibaba Developer - November 8, 2021
Alibaba Cloud Native Community - March 6, 2023
Xi Ning Wang(王夕宁) - July 21, 2023
Alibaba Cloud Native - November 3, 2022
Alibaba Cloud Community - March 9, 2022
A low-code development platform to make work easierLearn More
Help enterprises build high-quality, stable mobile appsLearn More
Multi-source metrics are aggregated to monitor the status of your business and services in real time.Learn More
Apsara Stack is a full-stack cloud solution created by Alibaba Cloud for medium- and large-size enterprise-class customers.Learn More
More Posts by OpenAnolis