EAS metrics - Platform For AI - Alibaba Cloud Documentation Center

After deploying an EAS service, use the monitoring page to track request volume, latency, and resource usage.

View monitoring data

Log on to the PAI console. Select a region on the top of the page. Then, select the desired workspace and click Elastic Algorithm Service (EAS).
Click a service name to open its details, then switch to the Monitoring tab.
View monitoring dashboards and metrics.
Switch Dashboards

In the upper-left corner of the Monitoring page, click the Service drop-down list to switch dimensions.
- Service: Service dimension. Default service monitoring dashboard name format is Service-<service_name>, where <service_name> is the EAS service name.
- Instance: Instance-level dashboard with single-instance and multi-instance views.
  - Single Instance: Metrics for one instance at a time. Switch between instances to compare.
    
    Select an instance from the Instance drop-down list on the right to view its monitoring metrics, such as QPS and RT.
  - Multiple Instance: Metrics for multiple instances side by side.
Switch Time Range

Click in the Monitoring area to change the dashboard time range.

In the time picker panel that opens, you can specify a custom start and end time under Absolute time range on the left, or select a preset range, such as Last 5 minutes to Last 7 days, from the Relative time ranges list on the right. The selected time range is marked with a check mark (✓).

Important
Minute-level metrics are retained for up to 1 month. Second-level metrics are retained for up to 1 hour.
Important
LLM monitoring metrics appear only when the service tag contains "ServiceEngineType": "vllm" or "ServiceEngineType" : "sglang".

Monitoring metrics

Service Monitoring Dashboard (Minute-Level)

This dashboard tracks the following metrics:

Metric		Description
QPS		Requests per second, grouped by return code. Summed across instances for multi-instance services. The 1d offset shows the previous day's QPS for trend comparison.
Response		Total responses in the selected time range, grouped by return code. Summed across instances for multi-instance services.
RT		Request response time. Avg: Average response time of all requests. TPXX: The XX-th percentile response time. For example, TP5 is the 5th-percentile max response time. TP100 is the max across all requests. For multi-instance services, TP100 is the max across all instances. Other TPXX values are averaged across instances.
Daily Invoke		Daily call count, grouped by return code. Summed across instances for multi-instance services.

More Metrics (CPU | Memory | GPU | Network | Resources)

Metric		Description
CPU	CPU	Average CPU usage of the service. Unit: CPU cores. Averaged across instances for multi-instance services.
	CPU Utilization	Average CPU utilization of the service. Calculation: Average CPU usage ÷ Maximum available CPU cores. Averaged across instances for multi-instance services.
	CPU Total	Total available CPU cores for the service. Calculation: Available CPU cores per single instance × Number of service instances.
Memory	Memory	Average memory usage of the service. Averaged across instances for multi-instance services. RSS: Resident physical memory size. Cache: Cache size. Total: Maximum available physical memory size for a single instance.
Memory	Memory Utilization	Average memory utilization of the service. Calculation: Memory RSS ÷ Memory Total. Averaged across instances for multi-instance services.
GPU	GPU Utilization	For GPU-enabled services, average GPU utilization. Averaged across instances for multi-instance services.
	GPU Memory	For GPU-enabled services, GPU memory usage. Averaged across instances for multi-instance services.
	GPU Total	For GPU-enabled services, total GPU amount. For services with multiple instances, this metric is the sum of GPUs across all instances.
	GPU Memory Utilization	For GPU-enabled services, GPU memory utilization. Averaged across instances for multi-instance services.
Network	Traffic	Traffic received and sent by the service, in bits per second. Averaged across instances for multi-instance services. Where: In: Traffic received. Out: Traffic sent.
Network	TCP Connections	Number of TCP connections.
Resources	Replicas	Number of service instances in different states: Total, Pending, Available.
Resources	Replicas By Resource	Number of service instances by resource type: Total, Dedicated (dedicated resources), Public (public resources).

Single Instance Monitoring Dashboard (Minute-Level)

This dashboard tracks the following metrics:

Metric	Description
QPS	Requests per second received by this instance. Grouped by return code.
RT	Response time of requests for this instance.
Response	Total responses received by this instance within the selected time range. Grouped by return code.

More Metrics (CPU | Memory | GPU | Network | Resources)

Metric		Description
CPU	CPU	CPU usage of this instance, in CPU cores.
CPU	CPU Utilization	Average CPU utilization of this instance. Calculation: Average CPU usage ÷ Maximum available CPU cores.
Memory	Memory	Memory usage of this instance. RSS: Resident physical memory size. Cache: Cache size. Total: Maximum available physical memory size for a single instance.
Memory	Memory Utilization	Average memory utilization of this instance. Calculation: Memory RSS ÷ Memory Total.
GPU	GPU Utilization	GPU utilization of this instance.
	GPU Memory	GPU memory usage of this instance.
	GPU Memory Utilization	GPU memory utilization of this instance.
Network	Traffic	Traffic received and sent by this instance, in bits per second. Where: In: Traffic received. Out: Traffic sent.
Network	TCP Connections	Number of TCP connections.

Multiple Instance Monitoring Dashboard

Minute-level and second-level metrics:

Minute-Level

Metric	Description
Instance QPS	Requests per second for each instance. Grouped by return code.
Instance RT	Average response time for each instance.
Instance CPU	CPU usage for each instance, in CPU cores.
Instance Memory -- RSS	Resident physical memory size for each instance.
Instance Memory -- Cache	Cache size for each instance.
Instance GPU	GPU utilization for each instance.
Instance GPU Memory	GPU memory usage for each instance.
Instance TCP Connections	Number of TCP connections for each instance.

Second-Level

Important

Data granularity is 5 seconds. Only the last 1 hour of data is retained.

Metric	Description
Instance QPS Fine	Requests per second received by each instance. Grouped by return code.
Instance RT Fine	Average response time for requests received by each instance.

GPU Monitoring Dashboard

GPU metrics at service and instance levels. Service-level values are averaged across all instances.

Metric	Description
GPU Utilization	GPU utilization of the service.
GPU Memory	GPU memory usage and total GPU memory of the service. Used: GPU memory usage. Total: Total GPU memory.
Memory Copy Utilization	GPU memory copy utilization of the service.
GPU Memory Utilization	GPU memory utilization of the service. Calculation: Memory usage ÷ Total memory.
PCIe	PCIe (Peripheral Component Interconnect Express) rate of the service, measured by DCGM. PCIe is a high-speed serial computer expansion bus standard. PCIe Transmit: PCIe transmission rate. PCIe Receive: PCIe reception rate.
Memory Bandwidth	GPU memory bandwidth metric of the service.
SM Utilization and Occupancy	SM (Streaming Multiprocessor) related metrics of the service. SMs are core components of a GPU, responsible for executing and scheduling parallel computing tasks. SM Utilization: SM utilization. SM Occupancy: Ratio of Warp threads residing on the SM.
Graphics Engine Utilization	GPU graphics engine utilization of the service.
Pipe Active Ratio	Activity rate of the GPU compute pipelines of the service. Pipe Fp32 Active Ratio: FP32 pipeline activity rate. Pipe Fp16 Active Ratio: FP16 pipeline activity rate. Pipe Tensor Active Ratio: Tensor pipeline activity rate.
Tflops Usage	Tflops (Tera floating-point operations per second) compute volume of the GPU compute pipelines of the service. FP32 Tflops Used: FP32 pipeline Tflops compute volume. FP16 Tflops Used: FP16 pipeline Tflops compute volume. Tensor Tflops Used: Tensor pipeline Tflops compute volume.
DRAM Active Ratio	Activity rate of the GPU device memory interface sending or receiving data.
SM Clock	SM clock frequency of the service.
GPU Temperature	GPU temperature related metrics of the service. GPU Temperature: GPU temperature. GPU Slowdown Temperature: GPU throttling temperature threshold. When the GPU temperature reaches this value, the GPU automatically reduces its operating frequency to prevent overheating. GPU Shutdown Temperature: GPU shutdown temperature threshold. When the GPU temperature reaches this value, the system forces the GPU device to shut down. This prevents hardware damage or more severe system failures due to GPU overheating.
Power Usage	GPU power consumption of the service.

The following are GPU health status and anomaly information metrics:

Metric	Description
GPU Health Count	Number of healthy GPU cards for the service.
GPU Lost Card Num	Number of lost GPU cards for the service.
ECC Error Count	Number of ECC errors for the service. ECC (Error Correction Code) detects and corrects errors during GPU memory data transmission or storage. Volatile SBE ECC Error: Number of single-bit volatile ECC errors for the service. Volatile DBE ECC Error: Number of double-bit volatile ECC errors for the service. Aggregate SBE ECC Error: Number of single-bit persistent ECC errors for the service. Aggregate DBE ECC Error: Number of double-bit persistent ECC errors for the service. Uncorrectable ECC Error: Number of uncorrectable ECC errors for the service.
NVSwitch Error Count	Number of NVSwitch errors for the service. NVSwitch provides high-bandwidth, low-latency communication channels for high-speed communication between multiple GPUs. NVSwitch Fatal Error: Number of fatal NVSwitch errors for the service. NVSwitch Non-Fatal Error: Number of non-fatal NVSwitch errors for the service.
Xid Error Count	Number of Xid errors for the service. Xid errors are error codes reported by the GPU driver. They indicate issues encountered by the GPU during operation. These errors are typically recorded in system logs (such as Linux dmesg or Windows Event Viewer) and represented as Xid codes. Xid Error: Number of non-fatal Xid errors for the service. Fatal Xid Error: Number of fatal Xid errors for the service.
Kernel Error Count	Number of non-Xid errors for the service. Non-Xid errors refer to other types of errors reported in kernel logs, excluding Xid errors.
Driver Hang	Number of GPU driver hangs for the service.
Remap Status	The state of the service when the GPU attempts to remap GPU memory rows.

VLLM Monitoring Dashboard

For multi-instance services, throughput metrics are summed and latency metrics are averaged across instances.

Metric	Description
Requests Status	Total requests for the service. Running: Number of requests running on the GPU. Waiting: Number of requests waiting for processing. Swapped: Number of requests swapped to the CPU.
Token Throughput	Number of input and generated tokens for all requests of the service. TPS_IN: Number of input tokens. TPS_OUT: Number of output tokens.
Request Completion Status	Completion status statistics for all requests of the service. preemptions: Requests preempted. stop: Requests successfully completed due to natural termination (the model output a stop token, such as <EOS>). length: Requests reached the maximum output token length. abort: Requests forcibly terminated.
Time To First Token	Time to first token latency for all requests of the service (time from receiving a request to generating the first token). Avg: Average time to first token latency for all requests. TPXX: Percentile values for time to first token latency for all requests.
Time Per Output Token	Time per output token latency for all requests of the service (average time required for each output token after the first token is generated). Avg: Average time per token latency for all requests. TPXX: Percentile values for time per token latency for all requests.
E2E Request Latency	End-to-end latency for all requests of the service (time from receiving a request to returning all tokens). Avg: Average end-to-end latency for all requests. TPXX: Percentile values for end-to-end latency for all requests.
Queue Time	Queue waiting latency for all requests of the service (time requests wait in queue for engine processing). Avg: Average queue waiting latency for all requests. TPXX: Percentile values for queue waiting latency for all requests.
Inference Time	Inference latency for all requests of the service (time requests are processed by the engine). Avg: Average inference latency for all requests. TPXX: Percentile values for inference latency for all requests.
Prefill Time	Prefill stage latency for all requests of the service (time the engine processes request input tokens). Avg: Average prefill latency for all requests. TPXX: Percentile values for prefill latency for all requests.
Decode Time	Decode stage latency for all requests of the service (time the engine generates output tokens). Avg: Average decode latency for all requests. TPXX: Percentile values for decode latency for all requests.
Input Token Length	Number of input tokens processed by the service. Avg: Average input token length for all requests. TPXX: Percentile values for input token length for all requests.
Output Token Length	Number of output tokens generated by the service. Avg: Average output token length for all requests. TPXX: Percentile values for output token length for all requests.
Request Parameters(params_n & max_tokens)	Parameter N and parameter max_tokens for all requests of the service. Params_n: Average value of parameter N for all requests. Params_max_tokens: Average value of parameter max_tokens for all requests.
GPU KV Cache Usage	Average GPU KV cache utilization of the service.
CPU KV Cache Usage	Average CPU KV cache utilization of the service.
Prefix Cache Hit Rate	Average prefix cache hit rate for all requests of the service. GPU: Average GPU prefix cache hit rate for all requests. CPU: Average CPU prefix cache hit rate for all requests.
HTTP Requests by Endpoint	Number of requests for the service, grouped by request method, path, and response status code.
HTTP Request Latency	Average latency for different request paths of the service.
Speculative Decoding Throughput	Speculative decoding count for the service. Averaged across instances for multi-instance services. Drafts: Number of Drafts Tokens generated. Draft Tokens: Number of Drafts Tokens processed. Accepted Tokens: Number of Drafts Tokens accepted. Emitted Tokens: Number of Drafts Tokens emitted.
Speculative Decoding Efficiency	Speculative decoding performance of the service. Draft Acceptance Rate: Average ratio of Drafts Tokens accepted. Efficiency: Average efficiency of speculative decoding.
Token Acceptance by Position	Number of Drafts Tokens accepted at different generation positions for the service. Averaged across instances for multi-instance services.

SGLang Monitoring Dashboard

For multi-instance services, throughput metrics are summed and latency metrics are averaged across instances.

Metric	Description
Requests Num	Total requests for the service. Running: Number of requests running on the GPU. Waiting: Number of requests waiting for processing.
Token Throughput	Number of input and generated tokens for all requests of the service. TPS_IN: Number of input tokens. TPS_OUT: Number of output tokens.
Time To First Token	Time to first token latency for all requests of the service. Time to first token latency is the time from receiving a request to generating the first token. Avg: Average time to first token latency for all requests. TPXX: Percentile values for time to first token latency for all requests.
Time Per Output Token	Time per output token latency for all requests of the service. Time per token latency is the average time required for each subsequent output token after the first token is generated. Avg: Average time per token latency for all requests. TPXX: Percentile values for time per token latency for all requests.
E2E Request Latency	End-to-end latency for all requests of the service. End-to-end latency is the time from receiving a request to returning all tokens. Avg: Average end-to-end latency for all requests. TPXX: Percentile values for end-to-end latency for all requests.
Cache Hit Rate	Average prefix cache hit rate for all requests of the service.
Used Tokens Num	Number of KV cache tokens used by the service. Averaged across instances for multi-instance services.
Token Usage	Average KV cache token utilization of the service. Averaged across instances for multi-instance services.

FAQ

LLM monitoring dashboard not displayed

Problem: After custom EAS deployment, the monitoring page shows only Service and GPU dashboards. LLM monitoring is missing.

Cause: The service configuration is missing the ServiceEngineType tag, which declares the inference engine type.

Note

Only the ServiceEngineType tag affects LLM monitoring. Other Model Gallery deployment parameters have no impact.

Solution: Update the service configuration to add the ServiceEngineType tag and set its value based on the inference engine used (only vllm or sglang are supported). For example:

{
  "labels": {
    "ServiceEngineType": "vllm"
  }
}

Q: Why do /metrics 200 frequently appear in logs?

Once the ServiceEngineType tag takes effect, the EAS backend polls the inference framework's /metrics endpoint every 10-15 seconds across all pods. This endpoint returns real-time metrics in Prometheus format for the LLM monitoring dashboard.

Q: How to check the maximum concurrency in monitoring?

EAS monitoring does not provide a direct "maximum concurrency" metric. You can determine concurrency based on the service type:

LLM inference services (vllm or sglang engine): Switch to the VLLM Monitoring Dashboard or SGLang Monitoring Dashboard, and check the Running metric under Requests Status or Requests Num. This metric indicates the number of requests currently running on the GPU, which represents the real-time concurrency. By changing the time range (for example, selecting Last 1 hours or Last 6 hours), you can find the peak value of the Running curve, which is the maximum concurrency within that period.
General services: You can estimate concurrency using the QPS (queries per second) and RT (response time) metrics from the service monitoring dashboard. Based on Little's Law: Concurrency ≈ QPS × Average RT (seconds). For example, if QPS is 10 and average RT is 2 seconds, the concurrency is approximately 20. Apply this formula during peak periods to estimate the maximum concurrency.

Reference

After enabling service monitoring alerts, you will receive alert notifications when the service triggers alert rules.
View EAS Cloud Monitor events through the Cloud Monitor console or API operation to perform O&M, auditing, or alert settings for these events.
Set custom monitoring metrics for Auto Scaling based on business logic. Custom Monitoring and Scaling Metrics.