Observability for vLLM and SGLang - Application Real-Time Monitoring Service

The Python agent of Application Monitoring now includes a vLLM/SGLang plugin that lets you observe vLLM and SGLang inference engines.

Note

Application Real-Time Monitoring Service (ARMS) currently supports observability only for the vLLM/SGLang framework.

Procedure

Step 1: Prepare environment variables

export ARMS_APP_NAME=xxx   # The name of the Elastic Algorithm Service (EAS) application.
export ARMS_REGION_ID=xxx   # The region ID for your Alibaba Cloud account.
export ARMS_LICENSE_KEY=xxx   # Your Alibaba Cloud license key.
export APSARA_APM_MODEL_SERVICE_FRAMEWORK=xxx # The type of the model-serving framework. Valid values: 1. vLLM-v0 2. vLLM-v1 3. SGLang

Step 2: Modify the inference engine command

Modify the run command for the inference engine.

The following example uses the DeepSeek-R1-Distill-Qwen-7B model.

Original vLLM command:

gpu_count=$(nvidia-smi --query-gpu=count --format=csv,noheader | wc -l);vllm serve /model_dir --host 0.0.0.0 --port 8000 --root-path '/' --trust-remote-code --gpu-memory-utilization 0.95 --max-model-len 32768 --tensor-parallel-size $gpu_count --served-model-name DeepSeek-R1-Distill-Qwen-7B

Modified vLLM command with ARMS observability:

gpu_count=$(nvidia-smi --query-gpu=count --format=csv,noheader | wc -l);export PIP_INDEX_URL=http://mirrors.cloud.aliyuncs.com/pypi/simple;export PIP_TRUSTED_HOST=mirrors.cloud.aliyuncs.com;export APSARA_APM_MODEL_SERVICE_FRAMEWORK=vLLM-v1;pip3 install aliyun-bootstrap;ARMS_REGION_ID=cn-hangzhou aliyun-bootstrap -a install;ARMS_APP_NAME=qwq32 ARMS_LICENSE_KEY=it0kjz0oxz@3115ad****** ARMS_REGION_ID=cn-hangzhou aliyun-instrument vllm serve /model_dir --host 0.0.0.0 --port 8000 --root-path '/' --trust-remote-code --gpu-memory-utilization 0.95 --max-model-len 32768 --tensor-parallel-size $gpu_count --served-model-name DeepSeek-R1-Distill-Qwen-7B

The modifications include the following steps:

Configure the PyPI repository. You can adjust this as needed.

export PIP_INDEX_URL=http://mirrors.cloud.aliyuncs.com/pypi/simple;export PIP_TRUSTED_HOST=mirrors.cloud.aliyuncs.com;

Configure the inference engine type.

export APSARA_APM_MODEL_SERVICE_FRAMEWORK=vLLM-v1;

Download the agent installer.
```
pip3 install aliyun-bootstrap;
```
Use the installer to install the agent.
Replace cn-hangzhou with your actual region.
```
ARMS_REGION_ID=cn-hangzhou aliyun-bootstrap -a install;
```

Original SGLang command:

python -m sglang.launch_server --model-path /model_dir

Modified SGLang command with ARMS observability:

export PIP_INDEX_URL=http://mirrors.cloud.aliyuncs.com/pypi/simple;export PIP_TRUSTED_HOST=mirrors.cloud.aliyuncs.com;export APSARA_APM_MODEL_SERVICE_FRAMEWORK=SGLang;pip3 install aliyun-bootstrap;ARMS_REGION_ID=cn-hangzhou aliyun-bootstrap -a install;ARMS_APP_NAME=qwq32 ARMS_LICENSE_KEY=it0kjz0oxz@3115ad****** ARMS_REGION_ID=cn-hangzhou aliyun-instrument python -m sglang.launch_server --model-path /model_dir

The modifications include the following steps:

Configure the PyPI repository. You can adjust this as needed.

export PIP_INDEX_URL=http://mirrors.cloud.aliyuncs.com/pypi/simple; export PIP_TRUSTED_HOST=mirrors.cloud.aliyuncs.com;

Configure the inference engine type.

export APSARA_APM_MODEL_SERVICE_FRAMEWORK=SGLang;

Download the agent installer.
```
pip3 install aliyun-bootstrap;
```
Use the installer to install the agent.
Replace cn-hangzhou with your actual region.
```
ARMS_REGION_ID=cn-hangzhou aliyun-bootstrap -a install;
```

Click Update.

Supported scenarios

ARMS supports only the official versions of vLLM and SGLang. Modified versions are not supported. For a list of supported versions, see LLM services.

Scenario	vLLM V0	vLLM V1	SGLang
Chat/Completion calls	Support	Supported	Supported
Embedding calls	Supported	Supported	Not supported
Rerank calls	Not supported	Supported	Not supported
KV Cache transfer duration	Not supported	Supported	Supported
Tokenizer/DeTokenizer execution duration	Not supported	Supported	Supported

FAQ

How do I configure the vLLM framework to include the TraceID in logs?

To include the TraceID in vLLM framework logs, set the following environment variables:

export OTEL_PYTHON_LOG_CORRELATION=true
export OTEL_PYTHON_LOG_LEVEL=info
export OTEL_PYTHON_LOG_FORMAT="%(asctime)s %(levelname)s [%(name)s] [%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s trace_sampled=%(otelTraceSampled)s] - %(message)s"
export VLLM_LOGGING_PREFIX="[%(filename)s:%(lineno)d] [trace_id=%(otelTraceID)s span_id=%(otelSpanID)s resource.service.name=%(otelServiceName)s trace_sampled=%(otelTraceSampled)s]"

Then, add the --enable-log-requests startup parameter to the vLLM framework's startup command to print request-level logs.

References

Spans and metrics collected for vLLM and SGLang: Provides a complete manifest of the collected data.
Collection configuration for vLLM and SGLang: Describes the configuration items for the collection switch, trace collection precision, input and output collection, metric collection, and engine troubleshooting.
Diagnose vLLM and SGLang applications with concurrency analysis: Introduces the concurrent analysis feature of the engine.