Deploy the OpenTelemetry Collector in a sidecar pattern within your function instance to automatically collect and report Prometheus metrics from the LLM inference engine, such as token throughput and request latency. You can connect these metrics to Application Real-Time Monitoring Service (ARMS) for Prometheus without changing your business code. This integration provides production-grade, real-time monitoring with visualization and alerting for your LLM service.
Preparations
Before you begin, prepare the required Alibaba Cloud services and resources, and obtain the necessary information and permissions for the configuration. If you have already completed these preparations, you can skip to the procedure.
Activate Object Storage Service (OSS) and create a bucket
Use a bucket to store model files, the OpenTelemetry Collector binary file, and its configuration file. This allows Function Compute (FC) instances to access them through mounting. If you already have a bucket, you can skip this section.
Activate Object Storage Service (OSS). In the navigation pane on the left, go to the Bucket List page and click Create Bucket.
Configure the following parameters. You can keep the default values for the other parameters.
Bucket Name: Enter a globally unique name. To ensure uniqueness, you can use a combination of the project name, region, and a random string, such as
llmfiles-hangzhou-a1b2c3.Region: Select the same region where you deployed the model service in Function Compute.
Click Finish.
Deploy an LLM inference model in Function Compute
If you do not have an LLM inference model service deployed in Function Compute, follow these steps to deploy a Qwen3-0.6B model that uses the vllm inference engine for testing. If you have already deployed a model, you can skip to the next section.
Download the model files and upload them to OSS. Download link: Qwen3-0.6B.
ImportantThe
model.safetensorsmodel file is large. Wait for all files to be successfully uploaded.Go to the Function Compute console and choose Function List. Select a region at the top of the page and click Create Function.
Select GPU Function and click Create GPU Function.
On the Create Function page, configure the following parameters. You can keep the default values for other parameters or configure them as needed. For more information, see Create a GPU-accelerated function.
Function Name: The name can contain only letters, digits, underscores (_), and hyphens (-). It cannot start with a digit or a hyphen. The name must be 1 to 64 characters in length. You can use a combination of the model name and a random string, such as qwen3-06b-a1b2c3.
Instance Type: Select Elastic Instance. Keep the default values for other settings.
Runtime Environment: Select Custom Image/Use image from custom repository.
Container Image:
serverless-registry.cn-hangzhou.cr.aliyuncs.com/functionai/vllm-openai:v0.10.1Start Command:
vllm serve /mnt/qwen3 --port 9000 --served-model-name Qwen/Qwen3-0.6B --trust-remote-codeListener Port: 9000.
Execution Timeout: Set this to 600.
Permissions: For Function Role, select AliyunFcDefaultRole.
Storage: Enable Mount OSS. For Bucket/Subdirectory, select the subdirectory that contains the model files. For Local Directory in Function, enter
/mnt/qwen3. This path must match the directory specified in the Start Command.
Click Create. You are redirected to the Function Details page.
In the Function Topology, click Trigger. Then, on the Edit Trigger page, set Authentication Method to No Authentication.
NoteSet Authentication Method to No Authentication for testing purposes only. For a production environment, authentication is required. For more information, see Authentication.
Activate ARMS and create a Prometheus instance
A Prometheus instance is used to receive and store performance metrics for the LLM model service that are pushed by the OpenTelemetry Collector. The instance also provides data query capabilities. If you do not have an available instance, follow these steps to create one and record its Remote Write Address. If you already have an instance, you can skip to the next section.
Activate Application Real-Time Monitoring Service and go to the ARMS console.
On the Prometheus Monitoring > Instance List page, click Create Prometheus Instance.
On the Create Instance page, set Instance Type to General-purpose Instance and specify a custom Instance Name. You can keep the default values for other settings or configure them as needed. For more information, see Manage a Prometheus instance.
Click Create. You are redirected to the Settings page of the instance. Record the Remote Write Address. You will need this address when you configure the OpenTelemetry Collector.
Procedure
Step 1: Download and configure OpenTelemetry Collector
The OpenTelemetry Collector collects Prometheus metrics from the LLM inference service and pushes them to a remote Prometheus instance using the Remote Write protocol.
Download and decompress otelcol-contrib_0.136.0_linux_amd64.zip. After decompression, you will obtain a binary file named
otelcol-contriband a configuration file namedotel_config.yaml.This topic uses version v0.136.0 as an example. For information about other versions, see opentelemetry-collector-releases
Modify the
otel_config.yamlconfiguration file using the following parameters. For a complete list of parameters, see opentelemetry/configuration.job_name: The job name. We recommend that you use the same name as the function to simplify metric queries and analysis.targets: Replace{port}with the listener port of the function. For example, the listener port for the function created in the Deploy an LLM inference model in Function Compute step is 9000.To find the port, go to the Function Management > Function List page. Click the function name to go to the Function Details page. You can find the port in the Configuration > Basic Configuration > Listener Port
username/password: Set these parameters to the AccessKey ID and AccessKey secret of your Alibaba Cloud account. If you use a RAM user, ensure that the user has the AliyunPrometheusMetricWriteAccess or AliyunCloudMonitorFullAccess permission. For more information, see Write data from open-source Prometheus to an Alibaba Cloud Prometheus instance using a remote write address.endpoint: Enter the public Remote Write address of the Prometheus instance.To find the address, go to the Prometheus Instance List. Click the name or ID of the instance. Copy the address from the settings page.
scrape_interval(Optional): The metric scraping interval.
The following code provides a complete example of the
otel_config.yamlconfiguration file:receivers: prometheus: config: scrape_configs: - job_name: "{user-vllm}" scrape_interval: 15s metrics_path: /metrics static_configs: - targets: ["0.0.0.0:9000"] processors: batch: extensions: basicauth/prw: client_auth: username: "LT******S2K" password: "t6Z7o******B55" exporters: prometheusremotewrite: endpoint: "https://workspace-default-******.cn-hangzhou.******/api/v1/write" auth: authenticator: basicauth/prw service: extensions: [basicauth/prw] pipelines: metrics: receivers: [prometheus] processors: [batch] exporters: [prometheusremotewrite]
Step 2: Upload the binary and configuration files to OSS
On the Bucket List page, click the name of the target bucket.
In the navigation pane on the left, choose File Management > File List, and then click Upload File.
In the upload panel, drag the
otelcol-contribbinary file and the modifiedotel_config.yamlfile from Step 1 into the upload area.Keep the default values for the other parameters and click Upload File.
Step 3: Modify the function instance configuration
Go to the Function Compute console. On the Function Management > Function List page, click the name of the target function.
Startup Parameter Configuration: By default, LLM metrics are enabled. For the vllm engine used in the example in this topic, this step is not required. For some engines, you must explicitly enable metrics using startup parameters.
For example, for SGLang, click the
Edit the settings in the Configuration > Advanced Configuration > Storage section. Enable Mount OSS and configure the OSS Mount Target:
Bucket/Subdirectory: Select the bucket and directory where the
otelcol-contribbinary file and the modifiedotel_config.yamlfile are located.NoteIf you have already mounted an OSS bucket as described in Deploy an LLM inference model in Function Compute, you can click Add Mount Target to add a new bucket.
Local Directory in Function: Enter the path of the local directory in the function instance to which you want to mount, such as
/mnt1/.Edit the settings in the Configuration > Instance Configuration section:
Enable Instance Prefetch.
Set Prefetch Timeout to 10.
The timeout period for the Initializer hook, in seconds. The value can range from 1 to 300. You can adjust this value as needed.
Set Prefetch Program Type to Execute Command. In the Command field, enter the following command:
NoteReplace `/mnt1` in the following command with the actual local directory that is mounted in the function.
nohup /mnt1/otelcol-contrib --config=/mnt1/otel_config.yaml >/dev/null 2>&1 &Click Deploy to complete the configuration.
Verification and testing
Verify that the model service is running correctly
Use a curl command to call the model service. Replace https://{fc_public_endpoint} with the public endpoint of the function's HTTP trigger. An example endpoint is https://*******.cn-****.fcapp.run.
To find the endpoint, go to the Function Management > Function List page. Click the function name to go to the Function Details page. You can find the public endpoint in the Triggers > Configuration > Public Endpoint
curl --request POST \
--url https://{fc_public_endpoint}/v1/chat/completions \
--header 'Authorization: Bearer REPLACE_BEARER_TOKEN' \
--header 'content-type: application/json' \
--data '{"messages":[{"content":"Hello! Who are you?","role":"user"}],"model":"Qwen/Qwen3-0.6B","stream":false}'Sample response:
{
"id":"chatcmpl-b27876a60f9749859638********b",
"object":"chat.completion",
"created":1762760556,
"model":"Qwen/Qwen3-0.6B",
"choices":[
{
"index":0,
"message":{
"role":"assistant",
"content":"It's a pleasure to meet all of you! I'm here to chat. We can talk about your studies, your life, or anything else.",
"refusal":null,
"annotations":null,
"audio":null,
"function_call":null,
"tool_calls":[],
"reasoning_content":null
},
"logprobs":null,
"finish_reason":"stop",
"stop_reason":null
}
],
"service_tier":null,
"system_fingerprint":null,
"usage":{
"prompt_tokens":14,
"total_tokens":42,
"completion_tokens":28,
"prompt_tokens_details":null
},
"prompt_logprobs":null,
"kv_transfer_params":null
} Verify that metrics are reported
Go to the ARMS console > Prometheus Monitoring > Instance List and click the name of the target Prometheus instance.
On the details page of the Prometheus instance, click Metric Management and select Metric Statistics. This page displays information such as the number of metrics and the data volume.
Click Explore Metrics and enter the following PromQL query:
vllm:prompt_tokens_created{}.
What to do next
Configure a Grafana dashboard
In the Application Real-Time Monitoring Service (ARMS) console, choose Grafana Service > Overview and click Create Workspace. Create a workspace that has the following configuration:
Configuration Item
Description
Workspace Name
A custom name for the Grafana workspace.
Admin Password
Set the Admin password.
Region
Select the region where the workspace is located.
Edition
The billing edition of Managed Service for Grafana. For the billing details of each edition, see Billing rules.
Grafana Version
Select a Grafana version.
Number of User Accounts
Select the number of accounts. For the billing details of each edition, see Billing rules.
Subscription Duration
Select a subscription duration.
Auto-renewal
If you select auto-renewal, the service will automatically enter a paid mode after the trial period expires. For the billing details of each edition, see Billing rules.
Resource Group
You can use resource groups to classify and manage resources under your Alibaba Cloud account. This lets you manage permissions, deploy resources, and monitor resources by group, rather than handling each resource individually.
Tags
Set a Tag Key and Tag Value for the workspace to facilitate group management.
In the Workspace List, select the workspace you just created.
In the Cloud Service Integration section, select Prometheus Monitoring Service to integrate the target Prometheus instance.
In the Basic Information section, click View Public Endpoint. Open the endpoint in a browser and select Sign in with Alibaba Cloud to log on using your Alibaba Cloud account.
Configure the visualization dashboard. You can use the Grafana configuration examples provided by vLLM, SGLang, and other services. The following content is for your reference.
Configure alerting
You can configure alerting in Alibaba Cloud Managed Service for Prometheus. For more information, see Create a Prometheus alert rule.
Billing
The main Alibaba Cloud services that are involved in this solution and their billing details are as follows:
Application Real-Time Monitoring Service (ARMS): The Prometheus and Grafana services that are used in this solution are provided by ARMS. For more information about billing, see Product billing (new).
Object Storage Service (OSS): Storing the
OpenTelemetry Collectorbinary and configuration files incurs a small storage fee. For more information about billing, see Billing overview.Function Compute (FC): Running the Collector process consumes a small amount of CPU and memory resources. You may need to adjust the function instance type based on the actual load. For more information about billing, see Billing overview.