Spans and metrics collected for vLLM and SGLang - Application Real-Time Monitoring Service

After the ARMS Python agent instruments a vLLM or SGLang inference engine, it reports spans, span attributes, span events, and metrics for the inference requests that the engine processes. Use this topic to identify the execution stage that each span in a trace represents, what each attribute measures, and how each metric is calculated. When a request slows down or a metric on your dashboard does not match your expectation, these descriptions help you analyze traces, build metric dashboards, and locate performance issues.

Common span conventions

The SpanKind of every span is INTERNAL. The spans that the agent generates express their business semantics through the gen_ai.span.kind attribute, which takes the value LLM, EMBEDDING, or RERANKER. The spans generated by native engine instrumentation do not carry this attribute, so you tell the stage from the span name.

Field name mapping across engines and versions

Fields that carry the same semantics use different names across engines, engine versions, and collection layers. The following table maps the equivalent field names:

Semantics	Field name	Reported by
Number of input tokens	`gen_ai.usage.input_tokens`	vLLM entry spans and SGLang entry spans
Number of input tokens	`gen_ai.usage.prompt_tokens`	vLLM V0 and V1 `llm_request` spans, and SGLang native request spans
Number of output tokens	`gen_ai.usage.output_tokens`	vLLM entry spans and SGLang entry spans
Number of output tokens	`gen_ai.usage.completion_tokens`	vLLM V0 and V1 `llm_request` spans, and SGLang native request spans
Number of candidates	`gen_ai.request.choice.count`	vLLM entry spans and SGLang entry spans
Number of candidates	`gen_ai.request.n`	vLLM V0 and V1 `llm_request` spans, and SGLang native request spans
Model name	`gen_ai.request.model`	vLLM entry spans and SGLang entry spans
Model name	`gen_ai.request.model_name`	vLLM V1 `llm_request` spans
Model name	`model_name` (metric dimension)	vLLM V1 engine-side metrics
Model name	`modelName` (metric dimension)	vLLM API-layer token metrics and SGLang token consumption metrics
Input tokens served from the cache	`gen_ai.usage.cache_read.input_tokens`	vLLM V1 `llm_request` spans and SGLang entry spans
Input tokens served from the cache	`gen_ai.usage.cached_tokens`	SGLang native request spans
Time to first token (metric)	`genai_llm_first_token_seconds`	vLLM metrics
Time to first token (metric)	`gen_ai_server_time_to_first_token`	SGLang metrics

vLLM spans

Span list

Span name	Produced at	Description
`chat {model name}`	OpenAI-compatible API layer	The entry point of a Chat Completions request, `/v1/chat/completions`.
`text_completion {model name}`	OpenAI-compatible API layer	The entry point of a Completions request, `/v1/completions`.
`embeddings {model name}`	OpenAI-compatible API layer	The entry point of an embedding request.
`rerank {model name}`	OpenAI-compatible API layer	The entry point of a rerank or score request.
`llm_request`	Inside the engine	The complete lifecycle of one request inside the engine. Produced by both V0 and V1.
`wait`	Inside the engine	The time a request waits in the queue to be scheduled. V1 only, collection granularity of 1 or higher.
`kv_transfer`	Inside the engine	The duration of pulling the KV cache from the remote side. Nested under `wait`. V1 only, collection granularity of 1 or higher.
`prefill`	Inside the engine	The duration of the Prefill stage. V1 only, collection granularity of 1 or higher.
`decode`	Inside the engine	The duration of the Decode stage. V1 only, collection granularity of 1 or higher.

kv_transfer is nested inside wait instead of being placed alongside it. While vLLM waits for the remote KV cache, the request stays in the WAITING_FOR_REMOTE_KVS state, so the transfer duration falls inside the waiting window.

Note

The real queuing time is the wait duration minus the kv_transfer duration.

In a PD-disaggregated deployment, a Prefill node produces only the prefill child span, and a Decode node produces only the decode child span.

Entry span attributes

The entry spans are generated by the general GenAI instrumentation, and their attribute definitions are consistent with the instrumentation of other large language model frameworks.

Attribute	Description
`gen_ai.span.kind`	The business semantic type. `LLM`, `EMBEDDING`, or `RERANKER`.
`gen_ai.operation.name`	The operation type. `chat`, `text_completion`, `embeddings`, or `rerank`.
`gen_ai.request.model`	The model name specified in the request.
`gen_ai.request.choice.count`	The number of candidates requested, which corresponds to the `n` parameter.
`gen_ai.request.temperature`	The sampling temperature.
`gen_ai.request.top_p`	The Top-P parameter.
`gen_ai.request.top_k`	The Top-K parameter.
`gen_ai.request.max_tokens`	The maximum number of tokens to generate.
`gen_ai.request.is_stream`	Whether the request is a streaming request.
`gen_ai.user.id`	The `user` field carried by the request.
`session.id`	The request identifier in streaming scenarios.
`gen_ai.response.id`	The response ID.
`gen_ai.response.model`	The model name returned in the response.
`gen_ai.response.finish_reasons`	The finish reasons.
`gen_ai.usage.input_tokens`	The number of input tokens.
`gen_ai.usage.output_tokens`	The number of output tokens.
`gen_ai.usage.total_tokens`	The total number of tokens.
`gen_ai.input.messages`	The input content. Controlled by the content collection switch.
`gen_ai.output.messages`	The output content. Controlled by the content collection switch.

`llm_request` span attributes (V0)

The unit of the duration attributes of a V0 engine is seconds.

Attribute	Description
`gen_ai.request.id`	The internal request ID of the engine.
`gen_ai.response.model`	The model name.
`gen_ai.request.temperature`	The sampling temperature.
`gen_ai.request.top_p`	The Top-P parameter.
`gen_ai.request.max_tokens`	The maximum number of tokens to generate.
`gen_ai.request.n`	The number of candidates.
`gen_ai.usage.num_sequences`	The number of sequences.
`gen_ai.usage.prompt_tokens`	The number of input tokens.
`gen_ai.usage.completion_tokens`	The number of output tokens.
`gen_ai.latency.e2e`	The end-to-end duration of the request.
`gen_ai.latency.time_in_queue`	The queuing duration.
`gen_ai.latency.time_to_first_token`	The time to first token.
`gen_ai.latency.time_in_scheduler`	The scheduling duration. Reported only when the engine provides this data.
`gen_ai.latency.time_in_model_forward`	The duration of the model forward computation. Reported only when the engine provides this data.
`gen_ai.latency.time_in_model_execute`	The model execution duration, including the forward computation, cross-worker synchronization, CPU-GPU synchronization, and sampling. Reported only when the engine provides this data.

`llm_request` span attributes (V1)

The unit of the duration attributes of a V1 engine is nanoseconds.

Attribute	Description
`gen_ai.request.id`	The internal request ID of the engine.
`gen_ai.request.model_name`	The model name.
`gen_ai.request.temperature`	The sampling temperature.
`gen_ai.request.top_p`	The Top-P parameter.
`gen_ai.request.max_tokens`	The maximum number of tokens to generate.
`gen_ai.request.n`	The number of candidates.
`gen_ai.usage.prompt_tokens`	The number of input tokens.
`gen_ai.usage.completion_tokens`	The number of output tokens.
`gen_ai.usage.cache_read.input_tokens`	The number of input tokens served from the cache without being prefilled again, including local prefix cache hits and remote KV reuse. Reported even when the value is 0, so that you can calculate the cache hit ratio.
`gen_ai.latency.e2e`	The end-to-end duration of the request.
`gen_ai.latency.time_in_queue`	The queuing duration.
`gen_ai.latency.time_to_first_token`	The time to first token.
`gen_ai.latency.time_in_model_prefill`	The duration of the Prefill stage.
`gen_ai.latency.time_in_model_decode`	The duration of the Decode stage.
`gen_ai.latency.time_in_model_inference`	The total duration of the inference stages (Prefill + Decode).
`gen_ai.latency.time_in_tokenize`	The prompt tokenization duration, which occurs in the frontend process.
`gen_ai.latency.time_in_detokenize`	The cumulative incremental detokenization duration.
`gen_ai.latency.time_in_kv_transfer`	The total duration of remote KV cache transfers. Reported only when a transfer occurred.
`gen_ai.request.preemption_count`	The number of times the request was preempted. Reported only when preemption occurred.
`gen_ai.engine.index`	The EngineCore instance index, which distinguishes engines in a multi-engine deployment.
`gen_ai.pd_role`	The PD disaggregation role. `producer` (Prefill node), `consumer` (Decode node), or `none` (not a PD-disaggregated deployment).

Note

Under PD disaggregation, only the Prefill duration is accurate on a Prefill node (time_in_model_decode is always 0), and only the Decode duration is accurate on a Decode node (time_in_model_prefill is always 0). vLLM does not reset the scheduling timestamp when it preempts a request, so the Prefill, Decode, and Inference durations already include the pauses caused by preemption. gen_ai.request.preemption_count is the evidence for explaining such duration spikes.

Stage child span attributes

The four child spans wait, kv_transfer, prefill, and decode all carry gen_ai.request.id, gen_ai.request.model_name, gen_ai.engine.index, and gen_ai.pd_role. kv_transfer carries the following additional attributes:

Attribute	Description
`gen_ai.kv_transfer.direction`	The transfer direction. The value is `recv`, which means the Decode node pulls the KV cache.
`gen_ai.kv_transfer.cached_tokens`	The number of tokens covered by this transfer, which is the number of tokens resolved on the remote side of the connector.
`gen_ai.kv_transfer.failed`	Whether the connector reported a load failure for this request.

Span events

When collection granularity is 2, span events are appended to llm_request for each scheduling iteration of the engine, which lets you reconstruct the execution process iteration by iteration.

Event name	Description
`WAIT`	The request enters the waiting state.
`PREFILL`	One Prefill iteration.
`DECODE`	One Decode iteration.
`PREEMPTED`	The request is preempted to release KV cache blocks.

Event attributes include gen_ai.engine.index and the token-level information gen_ai.token.ids, gen_ai.token.texts, gen_ai.token.count, and gen_ai.token.logprobs.

Note

The number of events is proportional to the number of iterations of the request and can easily exceed the OpenTelemetry default limit of 128 events per span, in which case events are dropped. Increase OTEL_SPAN_EVENT_COUNT_LIMIT when necessary.

vLLM metrics

Metric dimensions

The common dimensions carried by vLLM engine-side metrics:

Dimension	Description
`callType`	Always `gen_ai`.
`rpcType`	Always `2100`.
`engine_index`	The EngineCore instance index. V1 only.
`model_name`	The model name. V1 only.

gen_ai_vllm_request_success carries the additional finished_reason dimension. The API-layer token metrics carry the following dimensions:

Dimension	Description
`callType`	Always `gen_ai`.
`callKind`	Always `custom_entry`.
`rpcType`	Always `2100`.
`modelName`	The model name.
`spanKind`	`LLM` or `EMBEDDING`.
`usageType`	`input` or `output`.

Common metrics

V0 and V1 engines both report the following metrics:

Metric name	Type	Unit	Description
`genai_llm_usage_tokens`	Counter	Count	Input and output token consumption, distinguished by `usageType`.
`vllm_iter_count`	Counter	Times	The number of engine scheduling iterations.
`gen_ai_vllm_request_success`	Counter	Times	The number of successfully completed requests.
`genai_llm_first_token_seconds`	Counter	Seconds	The cumulative time to first token.
`gen_ai_server_time_per_output_token`	Counter	Seconds	The cumulative time per output token.
`gen_ai_server_request_duration`	Counter	Seconds	The cumulative end-to-end duration of requests.

V0 metrics

The system status metrics are gauges and reflect the instantaneous value at the collection moment.

Metric name	Type	Unit	Description
`gpu_cache_usage_sys`	Gauge	Ratio	GPU KV cache usage.
`cpu_cache_usage_sys`	Gauge	Ratio	CPU KV cache usage.
`num_running_sys`	Gauge	Count	The number of running sequences.
`num_waiting_sys`	Gauge	Count	The number of waiting sequences.
`num_swapped_sys`	Gauge	Count	The number of swapped-out sequences.
`num_prompt_tokens_iter`	Counter	Count	The number of input tokens processed in the iteration.
`num_generation_tokens_iter`	Counter	Count	The number of output tokens generated in the iteration.
`num_tokens_iter`	Counter	Count	The total number of tokens processed in the iteration.
`num_preemption_iter`	Counter	Times	The number of preemptions in the iteration.

Note

Gauge metrics are reported only by the processes that actually receive engine scheduling statistics. Processes that do not take part in inference, such as pure frontend processes, do not report 0 values, which avoids meaningless flat zero curves in a multi-process deployment.

V1 metrics

Metric name	Type	Unit	Description
`gen_ai_vllm_num_requests_running`	Gauge	Count	The number of requests in the running batch.
`gen_ai_vllm_num_requests_waiting`	Gauge	Count	The number of requests waiting to be processed.
`gen_ai_vllm_kv_cache_usage_perc`	Gauge	Ratio	KV cache usage, in the range of [0,1].
`gen_ai_vllm_prefix_cache_queries`	Counter	Count	The number of tokens queried in the prefix cache.
`gen_ai_vllm_prefix_cache_hits`	Counter	Count	The number of tokens that hit the prefix cache.
`gen_ai_vllm_num_preemptions`	Counter	Times	The cumulative number of engine preemptions.
`gen_ai_vllm_prompt_tokens`	Counter	Count	The number of tokens processed by Prefill.
`gen_ai_vllm_generation_tokens`	Counter	Count	The number of generated tokens.
`gen_ai_vllm_request_params_n`	Counter	Count	The cumulative value of the request `n` parameter.
`gen_ai_vllm_request_params_max_tokens`	Counter	Count	The cumulative value of the request `max_tokens` parameter.
`gen_ai_vllm_request_queue_time_seconds`	Counter	Seconds	The cumulative duration of requests in the WAITING stage.
`gen_ai_vllm_request_prefill_time_seconds`	Counter	Seconds	The cumulative Prefill stage duration of requests.
`gen_ai_vllm_request_decode_time_seconds`	Counter	Seconds	The cumulative Decode stage duration of requests.
`gen_ai_vllm_request_inference_time_seconds`	Counter	Seconds	The cumulative duration of requests in the RUNNING stage.

The prefix cache hit ratio is calculated as gen_ai_vllm_prefix_cache_hits divided by gen_ai_vllm_prefix_cache_queries.

SGLang spans

Span composition and hierarchy

A trace in an SGLang scenario consists of two kinds of spans: the entry spans that the agent generates at the OpenAI-compatible API layer, and the spans generated by native SGLang instrumentation. The agent injects the context of the entry span into the request header that it sends to the engine, so the native spans hang under the entry span and form one complete trace:

chat {model name}                             ← agent entry span
└─ Req {first 8 characters of the request ID} ← native request span, one per request
   ├─ Tokenizer (host:… | pid:…)              ← native thread span, frontend process
   │  ├─ tokenize
   │  └─ api_server_dispatch
   └─ Scheduler [TP 0] (host:… | pid:…)       ← native thread span, scheduler process
      ├─ prefill_waiting
      ├─ prefill_forward
      │  └─ chunked_prefill
      └─ decode_forward
         └─ decode_loop

Native instrumentation uses a three-level structure of request, thread, and stage. A request span represents one complete request. A thread span represents each process or thread that takes part in processing the request. A stage span represents one piece of processing logic inside that thread. In a multi-GPU deployment that uses tensor parallelism (TP), pipeline parallelism (PP), or data parallelism (DP), one request has multiple thread spans, one for each scheduler process. Span links are established between contiguous stage spans inside the same thread and between stage spans that connect across processes, which lets you reconstruct the execution order. A span link is not a parent-child relationship.

Entry span list

The entry instrumentation of the agent produces exactly one span for each request and generates no child spans. The stage breakdown inside a request is provided by native instrumentation.

Span name	Description
`chat {model name}`	A Chat Completions request, `/v1/chat/completions`.
`text_completion {model name}`	A Completions request, `/v1/completions`.

Entry span attributes

Request-side attributes:

Attribute	Description
`gen_ai.span.kind`	Always `LLM`.
`gen_ai.operation.name`	`chat` or `text_completion`.
`gen_ai.request.model`	The model name specified in the request.
`gen_ai.request.id`	The request ID. Reported for Chat streaming scenarios.
`gen_ai.request.is_stream`	Whether the request is a streaming request.
`gen_ai.request.temperature`	The sampling temperature.
`gen_ai.request.top_p`	The Top-P parameter.
`gen_ai.request.top_k`	The Top-K parameter.
`gen_ai.request.max_tokens`	The maximum number of tokens to generate.
`gen_ai.request.choice.count`	The number of candidates.

Response-side attributes:

Attribute	Description
`gen_ai.response.id`	The response ID.
`gen_ai.response.model`	The model name returned in the response. Reported for non-streaming scenarios.
`gen_ai.response.finish_reasons`	The finish reasons.
`gen_ai.usage.input_tokens`	The number of input tokens.
`gen_ai.usage.output_tokens`	The number of output tokens.
`gen_ai.usage.total_tokens`	The total number of tokens.
`gen_ai.usage.cache_read.input_tokens`	The number of input tokens served from the cache.
`gen_ai.latency.e2e`	The end-to-end duration of the request, in seconds.
`gen_ai.response.tokens_streamed`	The estimated number of output tokens in a streaming scenario, calculated as the number of streaming increments multiplied by the `stream_interval` of the engine. It fills in `gen_ai.usage.output_tokens` when the server does not return usage.
`gen_ai.input.messages`	The input content. Controlled by the content collection switch.
`gen_ai.output.messages`	The output content. Controlled by the content collection switch.

Tool calling attributes (Chat streaming scenarios):

Attribute	Description
`gen_ai.request.has_tool_calls`	Whether the request declares tools.
`gen_ai.request.tool_call_detected`	Whether a tool call is detected in the response.
`gen_ai.request.tool_calls_count`	The number of detected tool calls.
`gen_ai.completions.{i}.tool_calls.{j}.name`	The name of tool call `j` in candidate `i`.
`gen_ai.completions.{i}.tool_calls.{j}.arguments`	The arguments of that tool call.

Entry spans produce no custom span events. When the content collection policy is set to event, the input and output content is reported as a gen_ai.client.inference.operation.details log event instead of being written to span attributes.

Native instrumentation enablement and collection levels

The agent turns on the native tracing switch of SGLang when the engine starts, so you do not need to add --enable-trace manually. Native spans are reported through the global TracerProvider registered by the agent, so you also do not need to configure --otlp-traces-endpoint, and native spans belong to the same application as the entry spans.

Native instrumentation defines a collection level for each stage span, controlled by the SGLANG_TRACE_LEVEL environment variable. The default value 3 reports all stage spans at levels 1 to 3. A smaller value keeps only the more critical stages, and 0 turns off native instrumentation. The engine also provides the /set_trace_level?level=N operation for dynamic adjustment at runtime. The number of native spans grows with the deployment scale, and one request produces more than 10 spans in a multi-GPU deployment. When trace data volume is high, lower SGLANG_TRACE_LEVEL first.

Request spans and thread spans

The request span is the root span of native instrumentation. It is created and ended in the frontend process, and its name follows the format {role} Req {first 8 characters of the request ID}. The role is prefill or decode under PD disaggregation and is empty in a non-PD deployment.

Attribute	Description
`rid`	The request ID.
`module`	Always `sglang::request`.
`bootstrap_room`	The request correlation identifier under PD disaggregation. It is a hexadecimal string and is reported only in PD scenarios.
`gen_ai.request.id`	The request ID.
`gen_ai.request.temperature`	The sampling temperature.
`gen_ai.request.top_p`	The Top-P parameter.
`gen_ai.request.top_k`	The Top-K parameter.
`gen_ai.request.n`	The number of candidates.
`gen_ai.request.max_tokens`	The maximum number of tokens to generate.
`gen_ai.response.model`	The model name configured on the server.
`gen_ai.response.finish_reasons`	The finish reasons, as a JSON array string.
`gen_ai.usage.prompt_tokens`	The number of input tokens.
`gen_ai.usage.completion_tokens`	The number of output tokens. Reported for requests other than embedding requests.
`gen_ai.usage.cached_tokens`	The number of input tokens served from the cache.
`gen_ai.latency.time_to_first_token`	The time to first token.
`gen_ai.latency.e2e`	The end-to-end duration of the request.
`gen_ai.latency.time_in_model_prefill`	The duration from the completion of request dispatch to the first token.
`gen_ai.latency.time_in_model_decode`	The duration from the first token to the end of the request.
`gen_ai.latency.time_in_model_inference`	The duration from the completion of request dispatch to the end of the request.

The unit of all duration attributes is seconds. The attributes on a request span are filled in only when the request ends normally.

Note

The duration attributes are time differences measured from the perspective of the frontend process, and they are not exactly equivalent to the actual forward computation durations inside the scheduler. For the latter, see the actual durations of the stage spans.

The name of a thread span follows the format {thread label} [TP n] [PP n] [DP n] (host:{host identifier} | pid:{thread ID}). The rank information is appended only when the corresponding value exists. The thread label is Tokenizer, MultiTokenizer-{number}, DP Controller, or Scheduler, and it carries a Prefill or Decode prefix under PD disaggregation. The Detokenizer process has no native instrumentation, so no corresponding thread span appears.

Attribute	Description
`host_id`	The unique identifier of the host, which distinguishes processes that share the same PID on different nodes.
`pid`	The thread ID.
`thread_label`	The thread label.
`tp_rank`	The tensor parallelism rank. Reported when it exists.
`pp_rank`	The pipeline parallelism rank. Reported when it exists.
`dp_rank`	The data parallelism rank. Reported when it exists.

Note

When a request terminates abnormally, the error status and the end reason are recorded on the thread span: its status is set to ERROR and the attributes related to the end reason are appended. The request span itself is not marked as an error. When you investigate a failed request, check the status of the thread spans.

Stage span list

In the following tables, the Level column corresponds to SGLANG_TRACE_LEVEL. A span whose level value is larger than the configured value is not produced.

Common stages:

Span name	Level	Producing process	Description
`tokenize`	1	Frontend	The prompt tokenization duration.
`api_server_dispatch`	2	Frontend	The duration of dispatching the request from the frontend to the scheduler.
`dpc_dispatch`	2	DP Controller	The duration of the data parallelism controller dispatching the request. Produced only when DP is enabled.
`request_process`	2	Scheduler	The processing duration from the moment the request enters the scheduler to the moment it enters the waiting queue. Only in a non-PD-disaggregated deployment.
`prefill_waiting`	1	Scheduler	The time the request waits in the queue until it is scheduled for Prefill.
`prefill_forward`	1	Scheduler	The duration of the Prefill forward computation.
`chunked_prefill`	3	Scheduler	The Prefill duration of one chunk. Nested under `prefill_forward`.
`decode_waiting`	1	Scheduler	The wait between the completion of Prefill and the scheduling of the first Decode.
`decode_forward`	1	Scheduler	The overall duration of the Decode stage.
`decode_loop`	3	Scheduler	The duration of one Decode iteration. Nested under `decode_forward`.

Note

decode_forward has no independent end point. It is closed together with the thread when the request wraps up, so its duration covers the whole interval from the first Decode scheduling to the end of the request.

Speculative decoding stages, produced when speculative decoding is enabled and nested under the forward spans:

Span name	Level	Description
`spec_draft`	2	The duration of the draft model generating candidate tokens.
`spec_verify`	2	The duration of the target model verifying the draft tokens.
`spec_draft_extend`	3	The duration of the extend stage of the draft model.

Pipeline parallelism stages:

Span name	Level	Description
`run_batch_cpu`	4	The duration of CPU-side batch processing under pipeline parallelism. Level 4 is higher than the default value 3, so this span is produced only when you explicitly set `SGLANG_TRACE_LEVEL=4`.

Additional stages in a PD-disaggregated deployment, Prefill node:

Span name	Level	Description
`prefill_prepare`	1	The preparation duration after the Prefill node receives the request.
`prefill_bootstrap`	1	The duration of establishing the transfer connection with the Decode node.
`prefill_transfer_kv_cache`	1	The duration of transferring the KV cache to the Decode node.

Additional stages in a PD-disaggregated deployment, Decode node:

Span name	Level	Description
`decode_prepare`	1	The preparation duration after the Decode node receives the request.
`decode_bootstrap`	1	The duration of establishing the transfer connection with the Prefill node.
`decode_transferred`	1	The duration of waiting for the KV cache transfer to complete.
`fake_output`	3	The duration of the Decode node completing the output of the Prefill stage.

Note

Under PD disaggregation, Prefill and Decode each have an independent request span, which forms two independent traces. The two traces are correlated as one request through the bootstrap_room attribute on the request spans. On a Decode node in a PD-disaggregated deployment, decode_waiting changes its meaning to the queuing between the completion of the KV cache transfer and the start of the forward computation, which differs from its meaning in a non-PD deployment.

Stage span attributes and span events

Stage spans carry no custom attributes by default, with the following three exceptions:

Span name	Attribute	Description
`decode_loop`	`decode_ct`	The current Decode iteration number.
`spec_verify`	`num_correct_drafts`	The number of draft tokens that pass this verification. `accepted_tokens` is a compatible alias with the same value.
`run_batch_cpu`	`pp_mb_id`	The micro-batch number of pipeline parallelism.

Native instrumentation produces two kinds of span events:

Event name	Level	Attributes	Description
`retract`	1	None	Recorded when a request is rolled back and requeued because of insufficient resources.
`schedule`	3	`bid`, `batch_size`, `forward_mode`	Recorded when a request is placed into a batch. `bid` is the batch identifier, and `forward_mode` takes the value `decode`, `prefill`, or `prebuilt`.

Note

An event is attributed by timestamp to the stage span whose interval covers that moment. An event that falls outside every stage interval is attached to the thread span, so events with the same name can be attached at different positions in different requests.

SGLang metrics

Metric dimensions

The dimensions carried by engine-side metrics, which are scheduler and frontend statistics:

Dimension	Description
`callType`	Always `gen_ai`.
`callKind`	Always `custom_entry`.
`rpcType`	Always `2100`.

Token consumption metrics carry the additional dimensions modelName, spanKind, and usageType (input or output).

Metric list

Metric name	Type	Unit	Description
`genai_llm_usage_tokens`	Counter	Count	Input and output token consumption, distinguished by `usageType`.
`sglang_num_running_reqs`	Counter	Count	A snapshot of the number of running requests.
`sglang_num_queue_reqs`	Counter	Count	A snapshot of the number of requests in the queue.
`sglang_num_used_tokens`	Counter	Count	A snapshot of the number of tokens currently in use.
`sglang_token_usage`	Counter	Ratio	A snapshot of the KV cache token usage.
`sglang_gen_throughput`	Counter	Tokens/s	A snapshot of the generation throughput.
`gen_ai_sglang_cache_hit_rate`	Counter	Ratio	A snapshot of the cache hit ratio. Reported only when the engine provides this field.
`sglang_spec_accept_length`	Counter	Count	The accept length of speculative decoding.
`sglang_log_count`	Counter	Times	The number of statistical reports, which is used to calculate the average of the preceding snapshot metrics.
`prompt_tokens_total`	Counter	Count	The cumulative number of input tokens.
`generation_tokens_total`	Counter	Count	The cumulative number of output tokens.
`gen_ai_sglang_cached_tokens_total`	Counter	Count	The cumulative number of input tokens served from the cache.
`num_requests_total`	Counter	Times	The cumulative number of requests.
`gen_ai_server_time_to_first_token`	Counter	Seconds	The cumulative time to first token.
`gen_ai_server_time_per_output_token`	Counter	Seconds	The cumulative time per output token.
`sglang_inter_token_latency_seconds`	Counter	Seconds	The cumulative inter-token latency.
`gen_ai_server_request_duration`	Counter	Seconds	The cumulative end-to-end duration of requests.

Note

The system status metrics of SGLang are reported as counters, but semantically each value is an instantaneous snapshot of one statistical period. On a dashboard, average them within the period together with sglang_log_count instead of reading the cumulative value directly.

gen_ai_server_time_per_output_token is the end-to-end duration divided by the number of output tokens, so it is an average. To observe the real interval between tokens, use sglang_inter_token_latency_seconds.

Common span conventions

Field name mapping across engines and versions

vLLM spans

Span list

Entry span attributes

llm_request span attributes (V0)

llm_request span attributes (V1)

Stage child span attributes

Span events

vLLM metrics

Metric dimensions

Common metrics

V0 metrics

V1 metrics

SGLang spans

Span composition and hierarchy

Entry span list

Entry span attributes

Native instrumentation enablement and collection levels

Request spans and thread spans

Stage span list

Stage span attributes and span events

SGLang metrics

Metric dimensions

Metric list

`llm_request` span attributes (V0)

`llm_request` span attributes (V1)