After an EAS service is deployed, you can view metrics such as QPS, latency, error rate, and resource usage on the Monitoring tab of the service details page to understand the service invocation and running status.
View monitoring data
Access the monitoring page
Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Enter Elastic Algorithm Service (EAS).
Click the name of the target service to go to the service details page, and then switch to the Monitor tab.
Select dimensions and dashboards
The monitoring page provides two-level selection in the upper-left corner: first select a monitoring dimension (Service or Instance) from the drop-down list, and then select a dashboard.
LLM dashboards are available for LLM inference services that use the vLLM or SGLang engine. You must use an official PAI-EAS image or configure
ServiceEngineTypetosglangorvllmin the service labels. For more information, see LLM monitoring metrics > Prerequisites.GPU dashboards are available only for services that use GPUs.
Service dimension: View monitoring metrics for the entire service. After you select Service from the drop-down list, the following dashboards are available:
Service-<service_name>(where<service_name>is the name of the EAS service): the service monitoring dashboard. This dashboard displays general service monitoring metrics such as QPS, RT, CPU, and Memory.LLM: Displays service monitoring metrics based on the inference engine (SGLang or vLLM) used by the LLM service, such as request throughput and token usage.
GPU: Displays GPU hardware metrics such as SM utilization, memory bandwidth, and NVSwitch errors.
Instance dimension: View monitoring metrics for individual instances. After you select Instance from the drop-down list, the following dashboards are available:
Single Instance: The single-instance monitoring dashboard. You can select only one instance at a time. Only minute-level monitoring metrics are available.
Multiple Instance: The multi-instance monitoring dashboard. You can select multiple instances for comparison. Both minute-level and second-level monitoring metrics are available.
LLM Instance: The LLM instance monitoring dashboard. You can select one or more instances for comparison. This dashboard displays instance-level LLM inference metrics.
GPU Instance: The GPU instance monitoring dashboard. You can select one or more instances for comparison. This dashboard displays fine-grained GPU hardware metrics at the instance level.
The following table maps each dashboard to the corresponding metrics section.
Dashboard | Metrics section |
| |
Single Instance | Monitoring metrics > Single Instance Monitoring Dashboard (Minute-Level) |
Multiple Instance | |
GPU / GPU Instance | |
LLM / LLM Instance |
Change time range
Click 
on the right side of the Monitor area to change the time range of the dashboards.
In the time picker panel that opens, you can specify a custom start and end time (From / To) under Absolute time range on the left, or select a preset range, such as Last 5 minutes to Last 7 days, from the Relative time ranges list on the right. The selected time range is marked with a check mark (✓).
Minute-level metrics are retained for up to 1 month. Second-level metrics are retained for up to 1 hour.
Monitoring metrics
Service Monitoring Dashboard (Minute-Level)
This dashboard tracks the following metrics:
Metric | Description | |
QPS | Requests per second, grouped by return code. Summed across instances for multi-instance services. The 1d offset shows the previous day's QPS for trend comparison. | |
Response | Total responses in the selected time range, grouped by return code. Summed across instances for multi-instance services. | |
RT | Request response time.
| |
Daily Invoke | Daily call count, grouped by return code. Summed across instances for multi-instance services. | |
Single Instance Monitoring Dashboard (Minute-Level)
This dashboard tracks the following metrics:
Metric | Description |
QPS | Requests per second received by this instance. Grouped by return code. |
RT | Response time of requests for this instance. |
Response | Total responses received by this instance within the selected time range. Grouped by return code. |
Multiple Instance Monitoring Dashboard
Minute-level and second-level metrics:
Minute-Level
Metric
Description
Instance QPS
Requests per second for each instance. Grouped by return code.
Instance RT
Average response time for each instance.
Instance CPU
CPU usage for each instance, in CPU cores.
Instance Memory -- RSS
Resident physical memory size for each instance.
Instance Memory -- Cache
Cache size for each instance.
Instance GPU
GPU utilization for each instance.
Instance GPU Memory
GPU memory usage for each instance.
Instance TCP Connections
Number of TCP connections for each instance.
Second-Level
ImportantData granularity is 5 seconds. Only the last 1 hour of data is retained.
Metric
Description
Instance QPS Fine
Requests per second received by each instance. Grouped by return code.
Instance RT Fine
Average response time for requests received by each instance.
GPU Monitoring Dashboard
GPU metrics at service and instance levels. Service-level values are averaged across all instances.
Metric | Description |
GPU Utilization | GPU utilization of the service. |
GPU Memory | GPU memory usage and total GPU memory of the service.
|
Memory Copy Utilization | GPU memory copy utilization of the service. |
GPU Memory Utilization | GPU memory utilization of the service. Calculation: Memory usage ÷ Total memory. |
PCIe | PCIe (Peripheral Component Interconnect Express) rate of the service, measured by DCGM. PCIe is a high-speed serial computer expansion bus standard.
|
Memory Bandwidth | GPU memory bandwidth metric of the service. |
SM Utilization and Occupancy | SM (Streaming Multiprocessor) related metrics of the service. SMs are core components of a GPU, responsible for executing and scheduling parallel computing tasks.
|
Graphics Engine Utilization | GPU graphics engine utilization of the service. |
Pipe Active Ratio | Activity rate of the GPU compute pipelines of the service.
|
Tflops Usage | Tflops (Tera floating-point operations per second) compute volume of the GPU compute pipelines of the service.
|
DRAM Active Ratio | Activity rate of the GPU device memory interface sending or receiving data. |
SM Clock | SM clock frequency of the service. |
GPU Temperature | GPU temperature related metrics of the service.
|
Power Usage | GPU power consumption of the service. |
LLM monitoring metrics
LLM monitoring metrics are displayed in the LLM or LLM Instance dashboard regardless of whether the service uses the vLLM or SGLang engine. View the corresponding monitoring metrics based on the engine used.
Prerequisites
LLM dashboards are displayed when any of the following conditions are met:
An official PAI-EAS image is used.
ServiceEngineTypeis set tosglangorvllmin the service labels. Example:{ "labels": { "ServiceEngineType": "sglang" } }
Note:
The EAS backend calls the
/metricsendpoint of the engine every 10 to 15 seconds to collect Prometheus metrics. Therefore,/metrics 200entries appear regularly in the service logs.Minute-level metrics are retained for up to 1 month.
Multi-instance aggregation rules
For multi-instance services, throughput, count, and capacity metrics are summed across all instances. Ratio, utilization, and average latency metrics are averaged across all instances.
SGLang Monitoring Dashboard
VLLM Monitoring Dashboard
For multi-instance services, throughput metrics are summed and latency metrics are averaged across instances.
Metric | Description |
Requests Status | Total requests for the service.
|
Token Throughput | Number of input and generated tokens for all requests of the service.
|
Request Completion Status | Completion status statistics for all requests of the service.
|
Time To First Token | Time to first token latency for all requests of the service (time from receiving a request to generating the first token).
|
Time Per Output Token | Time per output token latency for all requests of the service (average time required for each output token after the first token is generated).
|
E2E Request Latency | End-to-end latency for all requests of the service (time from receiving a request to returning all tokens).
|
Queue Time | Queue waiting latency for all requests of the service (time requests wait in queue for engine processing).
|
Inference Time | Inference latency for all requests of the service (time requests are processed by the engine).
|
Prefill Time | Prefill stage latency for all requests of the service (time the engine processes request input tokens).
|
Decode Time | Decode stage latency for all requests of the service (time the engine generates output tokens).
|
Input Token Length | Number of input tokens processed by the service.
|
Output Token Length | Number of output tokens generated by the service.
|
Request Parameters(params_n & max_tokens) | Parameter N and parameter max_tokens for all requests of the service.
|
GPU KV Cache Usage | Average GPU KV cache utilization of the service. |
CPU KV Cache Usage | Average CPU KV cache utilization of the service. |
Prefix Cache Hit Rate | Average prefix cache hit rate for all requests of the service.
|
HTTP Requests by Endpoint | Number of requests for the service, grouped by request method, path, and response status code. |
HTTP Request Latency | Average latency for different request paths of the service. |
Speculative Decoding Throughput | Speculative decoding count for the service. Averaged across instances for multi-instance services.
|
Speculative Decoding Efficiency | Speculative decoding performance of the service.
|
Token Acceptance by Position | Number of Drafts Tokens accepted at different generation positions for the service. Averaged across instances for multi-instance services. |
FAQ
LLM monitoring dashboard not displayed
Problem: After custom EAS deployment, the monitoring page shows only Service and GPU dashboards. LLM monitoring is missing.
Cause: The service configuration is missing the ServiceEngineType tag, which declares the inference engine type.
Only the ServiceEngineType tag affects LLM monitoring. Other Model Gallery deployment parameters have no impact.
Solution: Update the service configuration to add the ServiceEngineType tag and set its value based on the inference engine used (only vllm or sglang are supported). For example:
{
"labels": {
"ServiceEngineType": "vllm"
}
}Q: Why do /metrics 200 frequently appear in logs?
Once the ServiceEngineType tag takes effect, the EAS backend polls the inference framework's /metrics endpoint every 10-15 seconds across all pods. This endpoint returns real-time metrics in Prometheus format for the LLM monitoring dashboard.
Q: How to check the maximum concurrency in monitoring?
EAS monitoring does not provide a direct "maximum concurrency" metric. You can determine concurrency based on the service type:
LLM inference services (vllm or sglang engine): Switch to the VLLM Monitoring Dashboard or SGLang Monitoring Dashboard, and check the Running metric under Requests Status or Requests Num. This metric indicates the number of requests currently running on the GPU, which represents the real-time concurrency. By changing the time range (for example, selecting Last 1 hours or Last 6 hours), you can find the peak value of the Running curve, which is the maximum concurrency within that period.
General services: You can estimate concurrency using the QPS (queries per second) and RT (response time) metrics from the service monitoring dashboard. Based on Little's Law:
Concurrency ≈ QPS × Average RT (seconds). For example, if QPS is 10 and average RT is 2 seconds, the concurrency is approximately 20. Apply this formula during peak periods to estimate the maximum concurrency.
Q: What is the difference between E2E Request Latency and RT?
E2E Request Latency is provided by the LLM inference engine, while RT is provided by the EAS engine (easworker). For LLM scenarios, E2E Request Latency is recommended.
Reference
After enabling service monitoring alerts, you will receive alert notifications when the service triggers alert rules.
View EAS Cloud Monitor events through the Cloud Monitor console or API operation to perform O&M, auditing, or alert settings for these events.
Set custom monitoring metrics for Auto Scaling based on business logic. Custom Monitoring and Scaling Metrics.