Propagate headers to trace gRPC services - Alibaba Cloud Service Mesh

Managed Service for OpenTelemetry provides a set of tools for distributed application development, including trace mapping, call statistics, trace topology, and application dependency analysis. Service Mesh (ASM) integrates with Managed Service for OpenTelemetry. This guide describes how to use headers to trace gRPC-based services in ASM, with language-specific implementations for Java, Go, Node.js, and Python.

How trace header propagation works

Each ASM sidecar generates a span when a request passes through it. Without application-level header propagation, each span appears as an isolated entry in Managed Service for OpenTelemetry and you lose the ability to visualize the full call chain.

To produce a connected trace, your application must:

Extract trace headers from each incoming request.
Forward those headers to all outgoing requests triggered by that incoming request.

This allows Managed Service for OpenTelemetry to correlate spans from different services into a single trace.

Prerequisites

An ASM instance is created. For more information, see Create an ASM instance.
Managed Service for OpenTelemetry is activated for your Alibaba Cloud account. For more information about billing, see Billing rules.

Sample project

The examples in this guide are based on the hello-servicemesh-grpc sample project (also available at hello-servicemesh-grpc). Clone the repository to follow along.

Extract headers on the gRPC server

On the server side, extract trace headers from each incoming request. The implementation varies by language and gRPC communication pattern.

Basic extraction methods

Language	API	Return type
Java	Implement `ServerInterceptor.interceptCall(ServerCall<ReqT, RespT> call, final Metadata m, ServerCallHandler<ReqT, RespT> h)`. Call `m.get(k)` where `k` is of type `Metadata.Key<String>`.	`String`
Go	`metadata.FromIncomingContext(ctx)`	`MD` (`map[string][]string`)
Node.js	`call.metadata.getMap()`	`{[key: string]: MetadataValue}` where `MetadataValue` is `string \| Buffer`
Python	`context.invocation_metadata()`	Array of `(key, value)` tuples. Access each entry with `m.key` and `m.value`.

Extraction by RPC pattern

The following table shows how to obtain the context or metadata object for each gRPC communication pattern. Java handles all patterns uniformly through the ServerInterceptor. Go, Node.js, and Python use the same API for all four patterns.

RPC pattern	Java	Go	Node.js	Python
Unary	`ServerInterceptor`	`metadata.FromIncomingContext(ctx)` -- `ctx` from the `Talk` method input	`call.metadata.getMap()`	`context.invocation_metadata()`
Server streaming	`ServerInterceptor`	`metadata.FromIncomingContext(stream.Context())` -- `stream` from `TalkOneAnswerMore` input	`call.metadata.getMap()`	`context.invocation_metadata()`
Client streaming	`ServerInterceptor`	`metadata.FromIncomingContext(stream.Context())` -- `stream` from `TalkMoreAnswerOne` input	`call.metadata.getMap()`	`context.invocation_metadata()`
Bidirectional streaming	`ServerInterceptor`	`metadata.FromIncomingContext(stream.Context())` -- `stream` from `TalkBidirectional` input	`call.metadata.getMap()`	`context.invocation_metadata()`

For Go streaming RPCs, obtain ctx from the stream parameter by calling stream.Context(). For unary RPCs, ctx is passed directly as a method parameter.

Inject headers from the gRPC client

Attach trace headers to each outgoing request so the downstream service receives them.

Basic injection methods

Language	API	Details
Java	Implement `ClientInterceptor.interceptCall(MethodDescriptor<ReqT, RespT> m, CallOptions o, Channel c)`. In the returned `ClientCall<ReqT, RespT>.start(Listener<RespT> l, Metadata h)`, call `h.put(k, v)` where `k` is `Metadata.Key<String>` and `v` is `String`.	Interceptor-based injection
Go	`metadata.AppendToOutgoingContext(ctx, kv...)`	Returns a new `context.Context` with appended key-value pairs
Node.js	`metadata = call.metadata.getMap()` then `metadata.add(key, headers[key])`	Add each header to the metadata map
Python	Build a dict (`metadata_dict[c.key] = c.value`) and convert with `list(metadata_dict.items())`	Pass the tuple list as metadata

Injection by RPC pattern

All four languages use the same injection method regardless of the RPC pattern. Java handles injection through the ClientInterceptor, Go calls metadata.AppendToOutgoingContext(ctx, kv), and Node.js and Python use their respective basic methods described above.

Propagate headers between server and client

A complete trace requires the server to pass received trace headers to the client before making downstream calls. The flow is:

Server extracts headers from the incoming request.
Server passes headers to the client-side code.
Client injects headers into the outgoing request.

Go, Node.js, and Python

The server-side method receives headers directly through the call context or metadata object. Pass the extracted headers to the client call within the same method scope -- no extra mechanism is needed.

Java: Metadata-Context propagation

Java uses two separate interceptors (ServerInterceptor and ClientInterceptor) to read and write headers. Because gRPC services may handle multiple requests concurrently, a simple shared cache cannot reliably connect the two interceptors.

Instead, Java uses Metadata-Context propagation:

Write to Context: In the ServerInterceptor, write the extracted headers into the gRPC Context: The key parameter is of type Context.Key<String>.
```
   ctx.withValue(key, metadata)
```
Read from Context: In the ClientInterceptor, read the headers back: By default, the get method uses Context.current(), which ensures the read and write operations share the same context.
```
   key.get()
```

This mechanism guarantees correct header propagation even when the server handles multiple concurrent requests.

Deploy and verify the ASM topology

Before you enable tracing, deploy and verify that the ASM instance topology works correctly.

The tracing directory in the sample project contains deployment scripts for Java, Go, Node.js, and Python. The following example uses Go.

cd go

# Deploy the topology
sh apply.sh

# Verify the topology
sh test.sh

If no errors occur, the topology is ready.

The following figure shows the deployed topology.

View tracing data

Configure ASM to send tracing data to Managed Service for OpenTelemetry. For more information, see Collect ASM tracing data to Managed Service for OpenTelemetry.
Log on to the Managed Service for OpenTelemetry console.
In the left-side navigation pane, click Trace Entrance.
On the Trace Entrance page, click Application Topology for the target application. The full trace chain is displayed: local requestor > Ingress Gateway > grpc-server-svc1 > grpc-server-svc2 > grpc-server-svc3.
Click the End-to-End Aggregation tab to view aggregated traces.
In the Span Name column, click a span to view its details.