Application Real-Time Monitoring Service:Distributed tracing concepts

Example: a distributed call

When a client sends a request, the call might flow like this:

1. The request arrives at a load balancer.

2. The load balancer forwards it to an authentication service and a billing service.

3. These services access the required resources.

4. The system returns a result to the client.

Figure 1. Example of a distributed call

A tracing system records every step as a span and assembles them into a trace. The trace is typically displayed as a timeline:

Figure 2. Trace diagram that contains a timeline

Span relationships

In the OpenTracing model, spans form a DAG through references. Two reference types exist:

The following trace contains eight spans (A through H) that illustrate both reference types:

Causal relationships among spans in a single trace


        [Span A]  <<<(root span)
            |
     +------+------+
     |             |
 [Span B]      [Span C] <<<(ChildOf: Span C is a child of Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F] >>> [Span G] >>> [Span H]
                                       ^
                                       |
                         (FollowsFrom: Span G follows Span F)

Here is the same trace as a timeline, showing how spans overlap:

Time relationships among spans in a single trace


--|-------|-------|-------|-------|-------|-------|-------|-> time

 [Span A***************************************************]
   [Span B**********************************************]
      [Span D******************************************]
    [Span C****************************************]
         [Span E*******]        [Span F**] [Span G**] [Span H**]

Reading the diagrams:

• Span A is the root span -- it has no parent and represents the entire trace duration.

• Span B and Span C are children of Span A (ChildOf). They run in parallel.

• Span D is a child of Span B.

• Span E is a child of Span C.

• Span F is also a child of Span C. Span G follows Span F (FollowsFrom), and Span H follows Span G.

What a span contains

Each span carries the following fields:

The following JSON shows what a typical span looks like in practice:

{
  "operationName": "/api/v1/order",
  "context": {
    "traceId": "7bba9f33312b3dbb8b2c2c62bb7abe2d",
    "spanId": "086e83747d0e381e"
  },
  "parentId": "",
  "startTime": "2025-06-15T08:30:01.209Z",
  "endTime": "2025-06-15T08:30:01.512Z",
  "tags": {
    "http.method": "POST",
    "http.status_code": 200,
    "component": "net/http"
  },
  "logs": [
    {
      "timestamp": "2025-06-15T08:30:01.305Z",
      "fields": { "event": "order.validated" }
    }
  ]
}

This is a root span -- it has a traceId and spanId but an empty parentId. A child span would carry the parent's spanId as its parentId.

OpenTelemetry uses different names for some of these fields. Tags are called attributes, and span logs are called span events. The underlying concepts are the same.

Tracer interface

The Tracer interface exposes three core methods:

Direct reporting

The application sends trace data directly to the Tracing Analysis backend.

Figure 3. Directly report data

Reporting through an agent

The application sends trace data to a local agent (for example, Jaeger Agent), which forwards it to the Tracing Analysis backend.

Figure 4. Report data by using an agent