In the cloud-native and microservices models, a production system often spans multiple language runtimes, such as Go, Java, Python, and Node.js. The deployment forms are scattered among containers, Kubernetes, and Serverless. To establish unified observability in such a heterogeneous environment, the traditional approach is to mount an intrusive Agent or SDK for each language. This involves modifying code, installing packages, aligning versions, and republishing. Each time a new service is connected, an engineering project is required. Under the fast research and development iteration pace, the cost of "connection means modification" is becoming increasingly unbearable.
At the same time, AI Agent applications are evolving from single Large Language Model (LLM) invocations to complex workflows with multi-step orchestration. A user request may trigger dozens of LLM invocations, Tool Calls, and AISearch. The invocation chain spans the Agent orchestration layer, LLM Provider, vector database, and external tools. Traditional application performance management (APM) is difficult to completely cover this. Zero-code observability solutions are equally indispensable in AI scenarios.
eBPF provides another idea. You can mount sandboxed probes in the Linux kernel to observe the network traffic, library function calling, and even system calls entering and leaving each process without modifying applications or restarting processes. Based on this capability, zero-code, cross-language, and low-overhead observability solutions are becoming a reality. OpenTelemetry eBPF Instrumentation (OBI) is the official answer provided by the OpenTelemetry community.
As an open source project officially maintained by OpenTelemetry, what OBI does can be summarized in one sentence. It utilizes the eBPF technology of the Linux kernel to automatically intercept and analyze network traffic and GPU operations entering and leaving the application without modifying any application code, and generates traces and metrics that comply with OpenTelemetry standards.
You can imagine it as a "X-ray glass" installed in the operating system kernel. Whether your application is written in Go, Java, Python, Node.js, or .NET, and regardless of what HTTP framework you use, what database you connect to, or which Large Language Model (LLM) you invoke, OBI can intercept communication at the kernel and library function layers, parse protocol semantics, and then output standard telemetry data. In terms of AI observability, OBI has built-in protocol-level tracking for four major GenAI Providers: OpenAI, Anthropic, Google Gemini, and Qwen. It can automatically detect LLM invocations and fetch Tool Call information from the response. It also supports the tracking of Rerank and AISearch operations, covering the core stages of the retrieval-augmented generation (RAG) pipeline.
It supports Linux amd64/arm64 architectures. The kernel requirement is 5.8+ (can be downgraded to 4.18+ for the RHEL series). The Deployment Method is flexible. You can run it as an independent process, deploy it as a Docker container, or deploy it as a Kubernetes DaemonSet.
The core capabilities of OBI revolve around three observability pillars.
The core competitiveness of OBI lies in protocol-aware probing. It not only records that "there is a network request", but also deeply understands the semantics of each request. The following is the complete protocol matrix currently supported.
| Protocol | TLS Support | Context propagation | Remarks |
|---|---|---|---|
| HTTP 1.x | Yes | Yes | Complete Request/response parsing |
| HTTP/2 | Yes | Go library level * | Automatic parsing of frames and header compression |
| gRPC | Yes | Go library level * | HTTP/2 frame-based method fetch |
| GraphQL | Yes | — | operation name and Type fetch |
Database
| Protocol | TLS | Featured capabilities |
|---|---|---|
| MySQL | Yes | SQL query text capture |
| PostgreSQL | Yes | SQL query text capture |
| MSSQL | Yes | SQL query text capture |
| Redis | Yes | db.namespace property |
| MongoDB | Yes | CRUD Operation Type detection |
| Couchbase | Yes | SQL++ query capture |
| Memcached | Yes | ASCII text-based protocol |
| Elasticsearch / OpenSearch | Yes | search/bulk/doc operation |
Message Queue
| Protocol | TLS | Featured capabilities |
|---|---|---|
| Kafka | Yes | produce/fetch, topic fetch |
| MQTT | — | 3.1.1 and 5.0, publish/subscribe |
| NATS | — | PUB/HPUB and MSG/HMSG |
| AWS SQS | Yes | Complete message operation |
How does OBI determine whether a segment of Transmission Control Protocol (TCP) payload is MySQL or Redis without decryption or relying on port conventions? The core lies in ReadTCPRequestIntoSpan (pkg/ebpf/common/tcp_detect_transform.go), which is a three-level waterfall match. It attempts to match in the order of "certainty from high to low" sequentially, and returns immediately upon a hit:
dispatchKernelAssignedProtocol, where the user mode directly switch event.ProtocolType. The kernel constants (common.go) are: MySQL=1, Postgres=2, Kafka=4, MQTT=5, MSSQL=6, NATS=7, AMQP=8. The SQL branch uses sentinel faults errFallback (fall back to the next level) or errIgnore (discard) for fine-grained control.detectGenericProtocol, which sequentially tries matchSQL → matchFastCGI → matchMongo → matchCouchbase → matchMemcached. SQL will first try the Request buffer, then the response buffer (when the response is hit, it invokes reverseTCPEvent to correct the direction).detectHeuristicProtocol, which tries matchRedis → matchMemcached → matchHTTP2 → matchNATS → matchAMQP → matchMQTT → matchKafkaFallback. The order itself is the accumulation of bug experience. For example, Hypertext Transfer Protocol 2 (HTTP/2) must be placed before Message Queuing Telemetry Transport (MQTT) because the heuristic of MQTT will falsely hit the connection preface of HTTP/2.Several details to prevent misjudgment that are worth following:
Each protocol has a corresponding TCPToToSpan constructor to form a request.Span.
The detection of OBI is divided into two layers. The first layer is language-independent network-level tracing—applications of any language can obtain basic traces and metrics through TCP traffic blocking. The second layer is runtime-specific depth Integration—for specific languages and frames, OBI directly hooks library functions through uprobe to achieve more precise context propagation and trace association.
| runtime | Integration method | depth capabilities |
|---|---|---|
| Go 1.17+ | uprobe directly hooks 13+ libraries | net/http, gin, gorilla/mux, gRPC, kafka-go, sarama, go-redis, MongoDB, database/sql, etc. Kernel reconstruction of goroutine parent-child relationship |
| Java 8+ | uprobe hooks OpenSSL / JVM key functions | TLS encrypted traffic plaintext tracking. Cross-process context goes through kernel tpinjector |
| Node.js 8+ | async_hooks uprobe | In-process asynchronous context propagation |
| .NET 6+ | OpenSSL uprobe + Network-level kprobe | TLS encrypted traffic plaintext tracking (protocol detection and cross-process propagation reuse the general Network Layer) |
| Python 3.9+ | asyncio / uvloop uprobe | coroutine context propagation |
| Ruby 3.0+ | Puma uprobe | rb_ary_shift work queue tracking |
| Nginx | Dedicated uprobe | HTTP server Request border detection |
| All languages | Network-level kprobe | TCP/HTTP automatic detection (language-independent baseline capability) |
The goroutine of Go drifts among operating system (OS) threads. The thread local storage of traditional application performance management (APM) completely fails. The solution of OBI is to rebuild the parent-child relationship of goroutines in the kernel (bpf/gotracer/go_runtime.c + go_common.h). The solution hooks runtime.newproc1 to record which goroutine creates which goroutine and writes the record into the least recently used (LRU) ongoing_goroutines. When an outbound invocation needs to find the inbound request to which the outbound invocation belongs, find_parent_goroutine traces back up to 6 layers upwards along the parent chain. The deep nest is to be compatible with Kafka clients such as franz-go. Then, the solution hooks runtime.casgstatus to trail the status switch and attach the OBI context to the goroutine. This allows the kernel probe (kprobe) on the same OS thread to associate correctly.
The asyncio event loop of Python alternately executes hundreds or thousands of coroutines (Tasks) on the same OS thread. The traditional assumption of "one thread corresponds to one request" completely fails. What is more troublesome is that asyncio.to_thread() delivers the work to the thread pool. These worker threads do not have the asyncio.Task identity at all. The solution of OBI is to track the Task and Context objects of CPython in the kernel and rebuild the parent-child relationship of the coroutines.
The core consists of four groups of user-space probes (uprobes) (bpf/generictracer/python.c). The task_step tracks which Task the event loop switches to. The _asyncio_Task___init__ records the parent-child relationship and inherits the request connection when the Task is created. The PyContext_CopyCurrent attaches the replica to the corresponding Task when the Context is copied. Both create_task and to_thread trigger this action. The context_run recovers the Task identity when the worker thread activates the Context. Three Berkeley Packet Filter (BPF) Maps (python_thread_state / python_task_state / python_context_task) work together to cover four concurrent modes: await, create_task, gather, and to_thread.
The parent chain traceback mechanism (find_python_owning_server_trace) is similar to Go. The mechanism traces upwards along the parent pointer from the current Task for up to 4 layers until the mechanism finds the ancestor Task that holds the inbound request connection (server_traces_aux). Then, the mechanism can associate with the correct server span. The problem of Task address reuse is solved by the version counter. Each time the Task is initialized, the version auto increments. The version is snapshotted when the Context is attached. If the comparison is inconsistent during the search, the Task is determined to expire.
The entire solution is anchored on the CPython _asyncio and libpython symbols, and does not depend on the internal implementation of uvloop. The uvloop only replaces the I/O driver of the event loop. The semantics of asyncio.Task and contextvars remains unchanged. Therefore, the same group of probes is effective for both asyncio and uvloop, and no additional adaptation is required.
The language runtime table in the previous text easily gives the impression that the cross-process context propagation is implemented by the language runtime of each language respectively. A more accurate expression is that the in-process context propagation is indeed exclusive to each language (Go goroutine traceback, Node async_hooks, Python asyncio, Ruby Puma queue, and Java/.NET tracking via OpenSSL/Java Virtual Machine (JVM) uprobe). However, the cross-process traceparent propagation is uniformly completed in the kernel mode by tpinjector for all non-Go languages (pkg/internal/ebpf/tpinjector + bpf/tpinjector/*.c). There are three methods:
sk_msg program rewrites the payload via the tail call chain and inserts the Traceparent: header. The SSL socket is directly skipped (the ciphertext cannot be modified).traceparent per stream, and use the huffman fingerprint 0x3fa9851d6b21834d to detect existing headers.bpf_store_hdr_opt writes trace_id/span_id in the WRITE_HDR_OPT callback. For inbound traffic, bpf_load_hdr_opt retrieves the trace_id/span_id in PARSE_ALL_HDR_OPT, and writes the trace_id/span_id into incoming_trace_map (using the normalization connection_info_t as the key) for Consumption by the server-side server_trace_parent before the trace_id/span_id is deleted. During startup, sock_iter.c also fills established persistent connections into sockhash, so that old connections can also be injected.Deployment notes: TCP Option kind=25 belongs to an unassigned IANA number. Some firewalls, Server Load Balancers, and cloud platform middleboxes may strip unknown TCP options, which causes the propagation to fail in silence. We recommend that you verify the pass-through capability of TCP options in the target network environment, or prioritize the use of the HTTP header injection method (OTEL_EBPF_BPF_CONTEXT_PROPAGATION=headers).
This is controlled by OTEL_EBPF_BPF_CONTEXT_PROPAGATION: headers/tcp/all/disabled. finder.go determines whether to load the injector based on this configuration.
How does a byte stream captured in the kernel become that trace in CloudMonitor 2.0? This is exactly the most hardcore engineering part of OBI. The user mode of OBI is not a large loop. Instead, the user mode is an explicitly declared, phased, and pluggable directed graph (DAG).
The entry point RunWithContextInfo (pkg/instrumenter/instrumenter.go) splits the three main pillars into three independent goroutines based on Feature Flags, and uses errgroup to attach them. If any goroutine crashes, the other two goroutines undergo a graceful shutdown when the context is canceled. The application observability pipeline is divided into three steps (pkg/internal/appolly/appolly.go): FindAndInstrument (Search and attach probes) → ReadAndForward (Start the processing pipeline) → WaitUntilFinished.
OBI has developed a minimalist node orchestration framework pkg/pipe/swarm. The core of the framework is the two-stage semantics of "instantiate all first, and then Run uniformly".
In the first stage, Instancer.Instance(ctx) invokes the InstanceFunc of each node in sequence. If an initialization failed error occurs, the framework immediately cancels the process and returns an error as a whole. No RunFunc will Start, which avoids the incomplete Status of a "half-Start". In the second stage, Runner.Start(ctx) spawns a goroutine for each node, which can be configured with WithCancelTimeout. After the context is canceled, if a node fails to exit within the timeout, Done() returns CancelTimeoutError and identifies the specific zombie node.
Nodes do not invoke each other directly. Instead, nodes pass messages through the generic queue msg.Queue[T] (pkg/pipe/msg/queue.go):
input.Bypass(output) directly transfers the upstream subscribers to the downstream nodes. The disabled node does not perform a dry-run, but physically disappears from the graph.SendCtx has a built-in sendTimeout timer (default 1 minute). An alerting is triggered if a channel write block of a subscriber times out. In PanicOnSendTimeout pattern, it directly panics and prints the A->B->C path.ClosingAttempts(n) + MarkCloseable() reference counting. It truly closes only after all producers are marked as closeable.The Build() function in pkg/internal/appolly/instrumenter.go explicitly pieces together the entire graph:
[per-process eBPF tracers]
| (processes share the Ring Buffer)
v
ringBufForwarder (reader goroutine + parser goroutine, object pool 2 × BatchLength)
| tracesInput (batch=100 / 1 s / 3 s idle-flush)
v
ReadFromChannel -> Routes -> KubeDecorator -> DockerDecorator -> NameResolution -> AttributesFilter
|
v exportableSpans ===== fan-out =====
|-- OTEL Traces Exporter
|-- Printer (debug)
|-- SpanNameLimiter -> [OTEL Metrics | SvcGraph Metrics | Prometheus]
`-- BPF MetricsTwo engineering design points: (1) The Metric sub-pipeline is started on demand. The setupMetricsSubPipeline is executed only if a Metric export is actually configured. (2) Special timeout for K8s decorators. The routerToKubeDecorator queue takes max(InformersSyncTimeout, ChannelSendTimeout) because the full informer snapshot must be pulled first during startup, and it cannot be mistakenly killed by the default deadlock detection.
This is the ingest endpoint of the entire pipeline, and also the most Performance-sensitive place (pkg/ebpf/common/ringbuf.go, generic ringBufForwarder[T], where T=Span on the application side and T=Record on the Stats side reuse the same Code):
readerLoop is only responsible for ReadInto the original record, and the parserLoop is responsible for parsing it into a Span. They pass the slot indices through two channels: freeIdx and workIdx.poolSize = 2 * BatchLength records for reuse (one batch is for the parser, and one batch is for concurrent padding by the reader).flushOnAvailableBytes actively flushes by checking the residual bytes of the ringbuf every 3 s to prevent Data from being stuck in the kernel under low traffic.SharedRingBuffer, and thousands of closers execute concurrent Close() upon exit.When any internal queue is blocked, OBI proactively prompts the full set of tuning knobs: OTEL_EBPF_OTLP_TRACES_BATCH_MAX_SIZE, OTEL_EBPF_OTLP_TRACES_QUEUE_SIZE, OTEL_EBPF_CHANNEL_BUFFER_LEN, OTEL_EBPF_CHANNEL_SEND_TIMEOUT, OTEL_EBPF_BPF_BATCH_LENGTH, and OTEL_EBPF_BPF_BATCH_TIMEOUT.
One-sentence summary: OBI = a swarm-orchestrated DAG + a set of fan-out queues with deadlock Detection + a dual-goroutine object pool ringbuf forwarder. The Start is either fully Succeeded or fully rolled back. The shutdown can locate zombie nodes. The disabled features physically disappear from the graph.
In addition to depth probing at the network protocol layer, OBI extends observability to the GPU Compute realm. By hooking libcuda.so via uprobes, it can track the core operations of NVIDIA CUDA: kernel launches, graph launches, memory assignments, and memory copies. This is highly valuable for GPU clusters running AI training and infer Jobs.
| CUDA operation | Tracking Data | Prometheus Metric |
|---|---|---|
cudaLaunchKernel |
grid dimension, block dimension | cuda_kernel_launch_calls_total, cuda_kernel_grid_size / block_size |
cudaGraphLaunch |
graph Start event | cuda_graph_launch_calls_total |
cudaMalloc |
Allocated memory size | cuda_memory_allocations_total |
cudaMemcpyAsync |
Copy size and direction | cuda_memory_copy_size |
The configuration of GPU tracking is very simple: OTEL_EBPF_CUDA_MODE=auto automatically detects whether the System has CUDA libraries and enables tracking.
GPU tracking uses uprobes to hook libcuda.so combined with an independent ring buffer (bpf/gpuevent on the kernel side, and pkg/internal/ebpf/gpuevent in user mode). An engineering detail worth pointing out is that CUDA events enter the processing pipeline through the same generic forwarder ringBufForwarder[T] (pkg/ebpf/common/ringbuf.go) as Network Spans. Therefore, CUDA events naturally share the same batch commit, backpressure control, and graceful shutdown mechanisms, eliminating the need to maintain separate transport logic for the GPU. This also explains why OTEL_EBPF_CUDA_MODE=auto can achieve zero-extra-configuration integration: it essentially just attaches an additional data source to the existing processing DAG.
In addition to application-level distributed tracking, OBI also provides Network-level observability.
Based on TC hooks, OBI catches L3/L4 network packets, parses IPv4/IPv6 and TCP/UDP headers, and generates network traffic Metrics. The Data pipeline supports rich decoration capabilities: Kubernetes metadata, reverse DNS, GeoIP geolocation, and Custom CIDR range annotation. This is suitable for intra-cluster and inter-cluster traffic Analysis, network topology visualization, and security audits.
By using kprobes or tracepoints, OBI collects node-level TCP RTT (obi_stat_tcp_rtt_seconds) and the Count of failed TCP connections (obi_stat_tcp_failed_connections) to help you monitor the underlying network quality.
The log enrichment of OBI is a language-independent capability. Regardless of what language or log frame the application uses, as long as it writes logs in JavaScript Object Notation (JSON) format to stdout/stderr or pipe, OBI can block them at the kernel layer and transparently inject trace_id and span_id. With zero modifications to the application Code, the Trace foreign key field automatically appears in the log file.
Implementation principle (bpf/logenricher/logenricher.c): Hook the tty_write (terminal Outputs) and pipe_write (pipe Outputs, covering the docker logs/kubectl logs scenarios) kernel functions via kprobe. When the write operation of the monitored process is triggered, the Berkeley Packet Filter (BPF) program reads the raw log Content (maximum 8 KB) from the user mode buffer. At the same time, it retrieves the trace context of the current thread or coroutine via obi_ctx, then uses bpf_probe_write_user to erase the raw log (padding with zero bytes), and sends the event via ringbuf to the user mode.
Known limitations: bpf_probe_write_user directly modifies the user mode memory, and there are two threats to follow: (1) A time window exists between the kernel zero-erasing and the user mode write-back. If the process crashes or the log collector happens to read the Outputs during this period, log loss or blank lines may occur. (2) Some security hardening kernels (such as those with the Lockdown pattern enabled) disable this function. Before deployment, you must confirm that the kernel configuration allows this operation.
After the user mode processor (pkg/internal/ebpf/logenricher) accepts the event: It attempts to parse the log line into JSON. If it is Succeeded, it injects the trace_id and span_id fields into the object (if they already exist, it does not overwrite them), and after it serializes them, it writes them back to the raw Outputs path of the application (TTY pts or pipe fd). If it is not JSON, it writes the log line back as is. The write-back path is parsed via path_resolver from the kernel file->f_path (TTY scenario) or located from /proc//fd/ (pipe scenario), and a Least Recently Used (LRU) cache is used to avoid repeated open operations.
Several engineering highlights: On the kernel side, the current fd is pre-recorded via ksys_write/do_writev kprobe, and when pipe_write is triggered, the file descriptor (fd) can be associated with the correct pipe. The asynchronous writer (ShardedQueue) performs shards by File Path, and the writing of the same file is Strict serial to guarantee the log line sequence. If the application already comes with the OpenTelemetry (OTel) software development kit (SDK) to export Traces (ExportsOTelTraces), only trace_id is injected, but not span_id, to avoid conflicts with the span of the application itself.
Via the Integration Center of CloudMonitor 2.0, select OpenTelemetry Non-intrusive Monitoring, and select one-click integration for the cluster [1].
After integration, you can view the number of Requests, Number Of Error, and Duration of the application, as well as the invoke details of the API.
In Trace Analysis, you can view the complete Trace link, making issues clear at a glance.
Taking the "source service → Target service" link as the core perspective, it displays the network traffic rate and TCP round-trip time (P50/P95/P99) between microservices in the Kubernetes (K8s) cluster in real-time. It is used for network quality inspection, delay abnormal localization, and traffic Distribution Analysis.
To better use the capabilities of OBI, you can achieve one-click integration via the Integration Center of CloudMonitor 2.0. Next, we will gradually supplement Artificial Intelligence (AI) Agent observability capabilities [2] and more Network Monitoring capabilities into OBI. You are welcome to build together.
[1] CloudMonitor 2.0 Integration Center: https://www.alibabacloud.com/help/cms/cloudmonitor-2-0/kan
[2] https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/issues/1854
741 posts | 60 followers
FollowAlibaba Cloud Native Community - January 8, 2026
Alibaba Cloud Native Community - April 16, 2026
Alibaba Cloud Community - December 21, 2021
Alibaba Cloud Native Community - May 8, 2026
Alibaba Cloud Native - June 21, 2024
Alibaba Cloud Native Community - November 6, 2025
741 posts | 60 followers
Follow
CloudMonitor
Automate performance monitoring of all your web resources and applications in real-time
Learn More
Microservices Engine (MSE)
MSE provides a fully managed registration and configuration center, and gateway and microservices governance capabilities.
Learn More
Cloud-Native Applications Management Solution
Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.
Learn More
EventBridge
EventBridge is a serverless event bus service that connects to Alibaba Cloud services, custom applications, and SaaS applications as a centralized hub.
Learn MoreMore Posts by Alibaba Cloud Native Community