Application Real-Time Monitoring Service:Diagnose vLLM and SGLang applications with concurrency analysis

Why you need a concurrency view

Large language model (LLM) inference is usually bottlenecked by memory bandwidth rather than compute. For an LLM, generating each token requires moving tens of GB of model weights from GPU memory into the compute units. The matrix multiplication of a single request uses only a tiny fraction of that compute, so the GPU spends most of its time waiting for data. Batching lets multiple requests share one weight read and spreads the memory access cost, which often increases single-GPU throughput several times over.

Static batching does not suit LLMs:

• Wasted slots — Output lengths vary widely within a batch, so a short request that has finished generating idles until the longest request ends, and its computed slot is wasted.

• Longer time to first token — Newly arrived requests must queue until the whole batch is released, which extends the time to first token (TTFT).

  Continuous Batching reduces the scheduling unit from an entire batch to a single iteration. After each decoding step, the engine checks the queue, immediately removes finished requests and releases their KV cache, and fills the freed slots with waiting requests. The batch stays saturated, the GPU no longer waits for long-tail requests, and both throughput and tail latency improve. This is why Continuous Batching is the default scheduling policy of modern inference engines such as vLLM and SGLang.

The trade-off is that troubleshooting a slow request cannot stop at its own stage durations. You also need the other requests that shared its batch at the same moment: how long their prompts are, whether they are in Prefill, and whether the number of concurrent requests has already reached the upper limit. Engine Concurrency Analysis plots this information on one timeline, which turns the batch environment of a request into something you can observe directly.

Prerequisites

• The ARMS Python agent is installed as described in Observability for vLLM and SGLang.

• The inference engine is started through the agent and reports trace data.

Limits

• Stage durations are required — The diagnostic methods in this topic depend on the Wait, Prefill, and Decode stage durations of a request.

• vLLM stage child spans — For vLLM, the wait, prefill, and decode stage child spans are produced only by V1 engines with a collection granularity of 1 or higher. The agent default is 1. When the collection granularity is set to 0, only the request-level llm_request span is produced. For the default value and the effective conditions of each collection switch, see Collection configuration for vLLM and SGLang.

Open Engine Concurrency Analysis

The concurrency analysis view is displayed per inference service application. Open the target application first, then switch to the Engine Concurrency Analysis tab and specify the time range to query.

1. Log on to the Cloud Monitor 2.0 console and select the target workspace. In the left-side navigation pane, choose All Features > Reasoning Services Observability, and then click the target inference service application.

2. In the top navigation bar, click Engine Concurrency Analysis.

3. In the time range picker in the upper-right corner of the page, set the time range that you want to query. The view displays execution snapshots of all requests in the engine within that time range.

Read the concurrency analysis view

The concurrency analysis view is an execution snapshot of all requests in the inference engine over a period of time. The horizontal axis is the timeline, and each block represents the execution interval of one inference request inside the engine. Pick a moment on the timeline and draw a vertical line: every block that the line crosses is an inference request running at that moment. Hover over a request to see the duration of each of its stages, including Wait, Prefill, and Decode.

The number of concurrent blocks in the view is limited by the maximum number of concurrent requests of the engine. For example, when the maximum number of concurrent requests is set to 8, at most 8 requests run at the same time, so the view shows at most 8 blocks. To check whether the engine is fully loaded at a given moment, count the blocks that the vertical line crosses.

Engine Concurrency Analysis supports both PD (Prefill/Decode) disaggregated deployments and non-PD-disaggregated deployments. The layout of the concurrency analysis view differs between the two deployment types:

• Non-PD-disaggregated deployment — All requests in the engine are plotted on one timeline.

• PD-disaggregated deployment — The Prefill workers and the Decode workers are displayed in separate panels, and the two stages of the same request are linked by a dashed line. Follow the dashed line to locate the corresponding Prefill request from the Decode side, and then evaluate the queuing duration and the execution duration on each side separately.

Diagnose a TPOT increase caused by a large Prefill blocking Decode

A large Prefill request blocks the Decode requests in the same batch. The Decode duration of that batch becomes noticeably higher than it was before the large Prefill request arrived, which shows up in metrics as an increase in time per output token (TPOT). Locate the blocking source in the concurrency analysis view as follows:

1. Find the time window in which the Decode duration rises, and align the extended Decode stages with the other requests in the same time window.

2. Compare the prompt token counts of these requests. If the prompt token count of one Prefill request is noticeably higher than that of the others, that large Prefill request is the main blocking source.

   In a PD-disaggregated deployment, follow the dashed line from the Decode side to locate the Prefill request that corresponds to each blocked Decode request, and then compare the prompt token counts.

Example — In a non-PD-disaggregated deployment with a maximum of 8 concurrent requests, one Prefill request has a prompt token count that is noticeably higher than that of the other requests in the same period. The duration of the Decode stages that overlap with this Prefill request rises, and the rise starts when that Prefill request enters the batch.

Diagnose request queuing caused by a concurrency spike

If the Wait stage of a request trace is unusually long, check the concurrency analysis view for the same period:

1. Find the time window in which the Wait stage is unusually long, and compare the number of concurrent requests in that time window with the number in earlier periods.

2. If the number of concurrent requests rises suddenly and long Wait stages appear at the same time, a traffic spike exceeded the processing capacity of the inference engine at that moment, and the new requests entered the queue.

3. Use the maximum number of concurrent requests to narrow down the cause. If the number of concurrent blocks stays at the limit for a long time, the engine capacity itself is insufficient. If it rises suddenly only within one time window, a traffic spike is the cause.

   In a PD-disaggregated deployment, evaluate the queuing duration on the Prefill side and the Decode side separately.