Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog What You Need to Know About eBPF Security Observability

What You Need to Know About eBPF Security Observability

This article highlights a speech from the PODS 2022 Conference, discussing the author’s insights on kernel security.

By Devin Xu, eBPF Technology Exploration SIG Maintainer & Linux Kernel Security Researcher in OpenAnolis.

This article shares the author's thoughts and insights on kernel security from the PODS 2022 Conference. This article was written based on three aspects: monitoring and observability, eBPF security observability analysis, and kernel security observability outlook.

1. Prospects for eBPF Security Observability

As shown in the figure below, monitoring is only the tip of the iceberg of observability. Most of the deep-seated problems hidden under the water cannot be solved by monitoring.

1

Monitoring vs. Observability

2

There is a trend of monitoring being visualized, but the vast majority of monitoring is done by defining parameters in advance and viewing logs afterward for analysis. The disadvantages of monitoring include:

  1. Poor scalability, requiring code modification and compilation, long validation cycles, and narrow data sources
  2. Observability is the collection of metrics and core data aggregation through active customization, including the following three types:

Logging

  • Real-time or post-event specific event information
  • Massive data traceability graph for distributed server clusters
  • A collection of discrete information and various asynchronous information

3

Tracing

  • Data Source: It provides data.
  • Collection Framework: It connects data sources first. Then, it collects, analyzes, and sends data. Later, it provides interfaces for user states.
  • Frontend Interaction: It connects to the kernel tracing frameworkand interacts with users directly. It is responsible for collecting configuration and data analysis.

4

Metrics

This is also the main difference between observability and monitoring.

Metrics refer to a certain type of information in the system, such as CPU, memory, network throughput, hard disk I/O, and hard disk usage. When the value of the metrics triggers the abnormal threshold, the system will send out alarm information or process actively, such as killing or isolating processes.

Purposes: Monitoring and alert

Let's conclude the difference between monitoring and observability:

Monitoring: It collects and analyzes system data, checks the current status of the system, and analyzes and handles foreseeable problems.

Observability: It means to observe the system and measure the internal status of the system and infer from the external output data that the system is in a certain degree of metrics at this time, especially the scenarios and events we care about.

Dimension Monitoring Observability
Scalability Scope Limitations Flexible and Extensive
Validation Cycle Long Cycle Immediate
Data Source Limited Area Customization
Observation Type Predictability Burstability

2. eBPF Security Observability Analysis

Security: It refers to the state that an object or the object attribute is not threatened.

Security Observability

  • By observing the entire system, from low-level kernel visibility to tracking file access, network activity or capability changes, all the way to the application layer, it covers such as function calls to vulnerable shared libraries, tracking process or parsing HTTP requests. So, security here should be considered a holistic concept.
  • It provides observability to various kernel subsystems, covering namespace escape, capabilities and privilege escalation, file system and data access, and network activities for protocols (such as HTTP, DNS, TLS, and TCP), with events at the system call layer to audit system calls and track process execution.
  • It checks the relevant output from the three dimensions of logging, tracing, and metrics to implement the capability of evaluating the internal security status of the system.

The security observability of eBPF is characterized by its extremely weak sense of existence to the kernel but extremely powerful observational capabilities:

  • Sandbox: It protects the stable operation of the kernel by using the eBPF Verifier.
  • Low Intrusiveness: No need to modify the kernel code and stop the program
  • Transparency: Data is collected transparently from the kernel to ensure the most important data assets of the enterprise.
  • Configurable: Customized and automated configuration policies (such as Cilium) have high update flexibility and richer filter conditions.
  • Fast Detection: Various events are directly processed in the kernel without returning to the user state, making abnormal detection convenient and fast.

Among them, the eBPF program is sandboxed and cannot be separated from the eBPF Verifier in safe mode (the most important of which is bound check):

  • It has the required privileges for the process of loading the eBPF program.
  • No crashes or other exceptions that cause the system to crash
  • The program can be terminated normally without a dead cycle.
  • Check whether the memory is out of bounds
  • Check register overflow

There are three main types of application scenarios for eBPF observability:

(1) Security Observability of Cloud-Native Containers

With the rapid development of cloud, edge, and end, people are increasingly focusing on the most popular cloud-native scenario currently. Falco, Tracee, Tetragon, Datadog-agent, and KubeArmor are several popular runtime protection solutions in cloud-native scenarios.

These solutions are mainly based on eBPF to mount kernel functions and write filtering policies. When an abnormal attack occurs at the kernel layer, predefined policies are triggered, directly issuing the alarms or even blocking operations without returning a message to the user layer.

Enforce security policies throughout the operating system in a preventative manner rather than reacting asynchronously to events. In addition to specifying, allow lists for multiple levels of access control. These solutions can also automatically detect privilege and capabilities escalation or namespace escalation (container escape) and automatically terminate affected processes.

Security policies can be injected through systems (such as Kubernetes (CRD), JSON API, or Open Policy Agent (OPA)).

(2) Security Observability Solutions on Application Layer

Such solutions are executed at the application and system call layer, and observability solutions vary. They all have a user space agent, but the agent relies on observability data collected by definition and then reacts to the data. Such solutions cannot observe kernel-level events.

5
6

(3) Security Observability Solutions on Kernel Layer

This type of solution is directly operated at the kernel layer, mainly for runtime reinforcement, and the observation capability is weak (or even null). The built-in kernel system provides a lot of policy execution options, but the kernel only focuses on providing access control capabilities when it is built, and it is very difficult to extend. For example, the kernel cannot be aware of Kubernetes and containers. Although the kernel module solves the scalability problem, it is often not a wise choice in many scenarios due to the security risks it generates.

The young kernel subsystem like LSM-eBPF is very powerful and promising, but it needs to rely on the latest kernel (≥ 5.7).

7
8

3. Kernel Security Observability Outlook

The following is a discussion of traditional kernel security, Android kernel security, and KRSI.

(1) Traditional Kernel Security Solutions

As Linus Torvalds once said, most security risks are caused by bugs, and bugs are part of the software development process; no bugs, no software.

As for whether it is a safe or non-safe vulnerability, the kernel community tests as much as possible and finds more potential vulnerabilities, which is similar to the blacklist approach.

The process of submitting kernel code is relatively complicated. When the kernel code is applied to a specific kernel version, there are problems of long cycle and version adaptation. Therefore, the development speed of kernel security is slower than other modules. At the same time, with the rapid development of intelligence, digitization, and cloudification, there are now tens of billions of devices based on Linux systems worldwide, and the security of these devices mainly depends on the security and robustness of the mainline kernel. When a kernel LTS version is issued with vulnerabilities, the related machines might be breached and exploited maliciously, and the loss is incalculable.

9

(2) Android Kernel Security

Nowadays, more smart terminals worldwide, including mobile phones, TV, SmartBox, IoT, cars, and multimedia devices, use the Android operating system, whose bottom layer is a Linux kernel. This also means the security of the Linux kernel has a significant impact on Android.

Due to historical reasons, Google's philosophy on the open-source of the Android kernel is not in line with the Linux kernel community. This has led to a number of specific modifications made to the Android kernel that cannot be incorporated into Upstream. It also leads to the fact that the Android kernel is partly different from the Linux kernel on the security side, and the focus is also different.

At the operating system level, the Android platform provides the security features of the Linux kernel and provides a secure inter-process communication (IPC) mechanism for secure communication between applications running in different processes. These security features at the operating system level are designed to ensure that even the native code is subject to application sandboxes. Whether the corresponding code is the result of self-contained application behavior or exploiting application vulnerabilities, the system can prevent illegal applications from harming other applications, the Android system, or the device itself.

10

Android kernel security features:

  • HWAddressSanitizer
  • KASAN
  • Top-Byte Ignore
  • KCFI
  • ShadowCallStack

(3) Kernel Security Observability Module-KRSI

The prototype of Kernel Runtime Security Instrumentation (KRSI) is implemented in the form of the Linux security module (LSM). The eBPF program can be mounted to the security hook of the kernel. The security of the kernel mainly involves inseparable two aspects: Signals and Mitigations.

  • Signals: This term means the system has some signs of abnormal activity or events.
  • Mitigations: This term refers to the alert or blocking actions taken after abnormal behavior is detected.

KRSI is implemented based on LSM, which means KRSI can make access control policy decisions. However, this is not the focus of KRSI's work. The focus is mainly to comprehensively monitor the system behavior to detect attacks. (It's the most important application scenario, but currently, it mainly does detection because it may be more dangerous to block processes rashly.) From this perspective, KRSI can be said to be an extension of the kernel audit mechanism, using eBPF to provide a higher level of configurability than the current kernel audit subsystem.

11

1) KRSI allows appropriate privileged users to mount BPF programs to any of the hundreds of hooks provided by the LSM subsystem.

2) KRSI has exported a new file system hierarchy under the /sys/kernel/security/bpf-each hook that corresponds to a file to simplify this step.

3) BPF programs (a new type of BPF_PROG_TYPE_LSM) can be mounted to these hooks through bpf() system calls, and multiple programs can be mounted to any given hook.

4) Whenever a security hook is triggered, all mounted BPF programs will be called in turn. As long as any BPF program returns an error state, the requested operation will be rejected.

5) KRSI can block operations from the function level, which is finer-grained and much less dangerous than processes.

(4) Follow-up Plan

  • Kernel security is a very complex topic in which one single move can affect the whole situation. It involves many challenging technologies, such as the defense mechanism, reinforcement configuration, and vulnerability exploitation. There will be related effects on performance or system stability in the process of reinforcement defense.
  • With the eBPF+LSM solution, the kernel security situation can be observed more visually with richer data. Then, we can have more development space for kernel Livepatch, vulnerability detection, and defense against related attack methods, such as privilege escalation.

The kernel security is crucial in that one single move can affect the whole situation, especially in terms of runtime safety. The new eBPF program mentioned above provides a unified API strategy for signals and mitigations, optimizes the kernel LSM framework, and solves the problem that in existing mechanisms, system calls are easy to be lost. From the perspective of blocking the operation of a function call, it realizes a finer-grained and more reasonable detection scheme. At the same time, it has further development space for kernel Livepatch, vulnerability detection, and defense against related attack methods, such as privilege escalation. The solution of eBPF combined with LSM is still evolving, and its functions and performance are gradually improving.

Everyone interested in the topic is welcome to join the eBPF Special Interest Group (SIG) of OpenAnolis. You are welcome to discuss and share your views and solutions on eBPF technology. Let’s start a magical journey of eBPF with SIG members!

eBPF SIG Address:
https://openanolis.cn/sig/ebpfresearch

Developers Observability eBPF extended Berkeley Packet Filter eBPF Security Kernel Security
1 2 1
Share on

Read previous post:

Kata 3.0 Is Coming! Start Experiencing the Out-of-the-Box Secure Container!

Read next post:

Virtio 1.2 is Coming!

OpenAnolis

83 posts | 5 followers

You may also like

Comments

Dikky Ryan Pratama May 6, 2023 at 12:32 pm

very easy article to understand.

OpenAnolis

83 posts | 5 followers

Related Products

More Posts by OpenAnolis

See All