Alibaba Cloud Observability and Datadog Release OpenTelemetry Go Automatic Instrumentation Tool

In the cloud-native observability realm, OpenTelemetry has become the de facto standard. Compared to Java, which possesses mature bytecode enhancement technology, the Go language, as a static compiled language, has long lacked a mature, low-intrusion automatic instrumentation solution. Current existing solutions mainly include:

eBPF: powerful but mainly leans towards the system call level. The processing of application-layer context (such as HTTP header propagation) is relatively complex.
Manual tracking: significant code changes and high maintenance costs. It requires modifying business code and the invocation methods of dependency libraries, explicitly adding traces and metrics logic at various key points.

To this end, the Alibaba Cloud Observability team and the Programming Language team explored the Go compile-time instrumentation solution and donated its core capabilities to the OpenTelemetry community, forming the opentelemetry-go-compile-instrumentation [1] project. With the joint efforts of companies such as Datadog and Quesma, we published the first Preview version V0.1.0 [2].

How it Works

The core of the automatic instrumentation tool lies in utilizing the -toolexec parameter of the Go compiler. The -toolexec intercepts the Go command and replaces it with our instrumentation tool. In this way, before the code is compiled, we have the opportunity to analyze and modify it. The entire process can be summarized into two phases:

1. Dependency Analysis

Before compilation starts, the tool analyzes the build flow of the application (go build -n) and detects third-party libraries used in the project, such as net/http, grpc, and redis. Then, it automatically generates a file named otel.runtime.go and imports the corresponding Hook code (monitoring logic, referred to as Hook code hereafter) into the build dependencies.

2. Code Injection

When the compiler processes the target function, the tool uses -toolexec to intercept the compilation, and then modifies the code of the target function. It inserts a segment of Trampoline Code at the function entry, and the Trampoline Code jumps to the pre-written Hook function.

● Before entering the function (Before): The Hook records the start time, fetches context information (such as HTTP headers), and starts a span.

● Function execution: Execute the original business logic.

● After exiting the function (After): The Hook catches the return value or Panic, ends the span, and records the duration.

The advantage of this method is zero runtime overhead (except for the necessary running time of monitoring logic). Because the instrumentation is directly compiled into the binary file, it does not require switching between Kernel space and user space, such as in eBPF, nor does it require loading at start, such as with a Java agent.

HTTP Instrumentation Example

Let's go through a simple HTTP example to see how it is used.

package main

import ...

func main() {
    http.HandleFunc("/greet", func(w http.ResponseWriter, r *http.Request) {
        w.Write([ ]byte("Hello, OpenTelemetry!"))
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Manual Instrumentation

You need to manually import the OpenTelemetry SDK, manually create a tracer, and manually start and end a span in the handler.

package main

import...

func initTracer() func(context.Context) error {
  /* ... dozens of lines of initialization code... */
}

func main() {
    // 1. Initialize the tracer.
    shutdown := initTracer()
    defer shutdown(context.Background())
    // 2. Wrap the handler.
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http. Request) {
        // 3. Manually extract the context and start the span.
        tracer := otel.Tracer("demo-server")
        ctx, span := tracer.Start(r.Context(), "GET /greet")
        // 4. Ensure that the span ends.
        defer span.End()
        // 5. You may also need to manually record attributes.
        span.SetAttributes(attribute.String("http.method", "GET"))
        w.Write([]byte("Hello, OpenTelemetry! "))
        })
        // 6. ListenAndServe may also need to be wrapped...
        log.Fatal(http.ListenAndServe(":8080", handler))
}

For microservices with hundreds or thousands of APIs, the cost of this modification is catastrophic.

Automatic Instrumentation

Download tool: You can go to the Release Page [2] to download.
Compile application: ./otel-linux-amd64 go build -o myapp
Configure and run: export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" export OTEL_SERVICE_NAME="my-app" ./myapp

The compiler will silently "weave" the monitoring logic of HTTP requests into the application binary file. After the OpenTelemetry export endpoint (such as Jaeger or console) is configured, you can run the generated server. When the /greet API is accessed, tracing data is automatically generated and reported, containing information such as the URI of the request, duration, and status code.

From Commercial to Open-source

During the deep practice of eBPF technology, although we acknowledge its power, we also found that it is difficult to perfectly handle the application layer context. More importantly, we continuously received user feedback that everyone is troubled by the tedious manual tracking and high maintenance costs.

To solve this pain point, we began to explore the Go compile-time automatic instrumentation solution, published it to Application Real-Time Monitoring Service (ARMS) of Alibaba Cloud Observability [3], continually iterated in this most rigorous "experimental field," and gradually evolved it into a mature solution. It not only achieves tracing analysis with zero code modification but also extends support for rich advanced features such as metric statistics, runtime monitoring, and even continuous profiling. It can even complete the event tracking of the enterprise internal SDK via custom extension features [4].

Tracing analysis

Continuous profiling

This solution has been successfully verified by customers in many realms such as e-commerce, short plays, AI video, and automotive. After seeing the immense value it brings to users and verifying its stability and feasibility, we decided to contribute its core capabilities to the OpenTelemetry community, hoping it will become an inclusive technology. At the same time, we collaborated with Datadog, a top vendor in the observability realm, to jointly promote and finally facilitate the birth of this official project [1].

Currently, the project is in an active development stage. We welcome everyone to try it out, provide feedback, and participate in contributions to jointly build a better cloud-native observability ecosystem.

[1] OpenTelemetry Go compile instrumentation project: https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation

[2] Release link: https://github.com/open-telemetry/opentelemetry-go-compile-instrumentation/releases/tag/v0.1.0

[3] Alibaba Cloud ARMS Go agent Commercial Edition: https://www.alibabacloud.com/help/en/arms/application-monitoring/user-guide/monitoring-the-golang-applications/

[4] Custom extension: https://www.alibabacloud.com/help/en/arms/application-monitoring/use-cases/use-golang-agent-to-customize-scalability

Community

Alibaba Cloud Observability and Datadog Release OpenTelemetry Go Automatic Instrumentation Tool

How it Works

1. Dependency Analysis

2. Code Injection

HTTP Instrumentation Example

Manual Instrumentation

Automatic Instrumentation

From Commercial to Open-source

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

DevOps Solution

Alibaba Cloud Flow

Managed Service for OpenTelemetry

Managed Service for Prometheus