exports metrics and logs for generative AI (GenAI) requests that follow the OpenTelemetry (OTel) GenAI Semantic Conventions. This topic describes how to use to output these metrics and logs.
Background information
The OpenTelemetry (OTel) GenAI Semantic Conventions are a set of standardized guidelines for monitoring and tracing applications that use generative AI, such as those based on a Large Language Model (LLM), text generation, or image generation. The goal is to unify metrics, logs, and trace data for generative AI requests to simplify cross-system analysis and troubleshooting. The conventions' core objectives are:
-
Standardize data collection:
Define common attributes for generative AI requests, such as model names, input and output token counts, and configuration parameters.
-
Enable end-to-end tracing:
Correlate generative AI requests with trace data from other systems, such as databases and API gateways.
-
Unify analysis and monitoring:
Define standardized tags that enable tools like Prometheus and Grafana to aggregate and visualize data.
Prerequisites
-
Gateway with Inference Extension version 1.4.0 or later installed, with Enable Gateway API Inference Extension (Requires a deployed inference service) selected during installation (see Install components)
-
The mock-vllm application is deployed.
Configure observability data output
Deploy the generative AI observability plugin
requires the gen-ai-telemetry observability plugin to export observability data. The plugin is provided as a container image and does not have a fixed release cadence. You can find the latest image version in the gen-ai-telemetry plugin release history.
kubectl apply -f - <<EOF
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
name: ack-gateway-llm-telemetry
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: mock-route
wasm:
- name: llm-telemetry
rootID: ack-gateway-extension
code:
type: Image
image:
url: registry-cn-hangzhou.ack.aliyuncs.com/acs/gen-ai-telemetry-wasmplugin:g76f5a66-aliyun
EOFYou can pull the gen-ai-telemetry observability plugin image over a private network. If your cluster cannot pull the image from the public internet, you can replace the image URL with the VPC endpoint for your region. For example, if your cluster is in the China (Beijing) region, use registry-cn-beijing-vpc.ack.aliyuncs.com/acs/gen-ai-telemetry-wasmplugin:{image_tag} to pull the image.
Configure gateway metrics tag rules
When you deploy the mock-vllm application, an EnvoyProxy resource named custom-proxy-config is created. To output metrics data for the egress gateway, you need to add Metrics Tag rules to this resource.
-
Edit the EnvoyProxy resource.
kubectl edit envoyproxy custom-proxy-config -
Update
custom-proxy-configwith thespec.bootstrapcontent from the following YAML.apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: custom-proxy-config namespace: default spec: bootstrap: type: JSONPatch jsonPatches: - op: add path: /stats_config value: stats_tags: - tag_name: gen_ai.operation.name regex: "(\\|gen_ai.operation.name=([^|]*))" - tag_name: gen_ai.system regex: "(\\|gen_ai.system=([^|]*))" - tag_name: gen_ai.token.type regex: "(\\|gen_ai.token.type=([^|]*))" - tag_name: gen_ai.request.model regex: "(\\|gen_ai.request.model=([^|]*))" - tag_name: gen_ai.response.model regex: "(\\|gen_ai.response.model=([^|]*))" - tag_name: gen_ai.error.type regex: "(\\|gen_ai.error.type=([^|]*))" - tag_name: server.port regex: "(\\|server.port=([^|]*))" - tag_name: server.address regex: "(\\|server.address=([^|]*))"After you save and exit, the configuration takes effect immediately. The gateway can now export generative AI metrics.
Configure log output
To export gateway logs, you also need to modify the EnvoyProxy resource. Add the appropriate configuration based on your requirements.
-
Edit the EnvoyProxy resource.
kubectl edit envoyproxy custom-proxy-config -
Update the
spec.telemetrycontent from the following YAML intocustom-proxy-config.apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyProxy metadata: name: custom-proxy-config namespace: default spec: telemetry: accessLog: disable: false settings: - sinks: - type: File file: path: /dev/stdout format: type: JSON json: # Default access log fields start_time: "%START_TIME%" method: "%REQ(:METHOD)%" x-envoy-origin-path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" protocol: "%PROTOCOL%" response_code: "%RESPONSE_CODE%" response_flags: "%RESPONSE_FLAGS%" response_code_details: "%RESPONSE_CODE_DETAILS%" connection_termination_details: "%CONNECTION_TERMINATION_DETAILS%" upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%" bytes_received: "%BYTES_RECEIVED%" bytes_sent: "%BYTES_SENT%" duration: "%DURATION%" x-envoy-upstream-service-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" user-agent: "%REQ(USER-AGENT)%" x-request-id: "%REQ(X-REQUEST-ID)%" :authority: "%REQ(:AUTHORITY)%" upstream_host: "%UPSTREAM_HOST%" upstream_cluster: "%UPSTREAM_CLUSTER%" upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%" downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%" downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%" requested_server_name: "%REQUESTED_SERVER_NAME%" route_name: "%ROUTE_NAME%" # Fields for generative AI requests gen_ai.operation.name: "%FILTER_STATE(wasm.gen_ai.operation.name:PLAIN)%" gen_ai.system: "%FILTER_STATE(wasm.gen_ai.system:PLAIN)%" gen_ai.request.model: "%FILTER_STATE(wasm.gen_ai.request.model:PLAIN)%" gen_ai.response.model: "%FILTER_STATE(wasm.gen_ai.response.model:PLAIN)%" gen_ai.error.type: "%FILTER_STATE(wasm.gen_ai.error.type:PLAIN)%" gen_ai.prompt.tokens: "%FILTER_STATE(wasm.gen_ai.prompt.tokens:PLAIN)%" gen_ai.completion.tokens: "%FILTER_STATE(wasm.gen_ai.completion.tokens:PLAIN)%" gen_ai.server.time_per_output_token: "%FILTER_STATE(wasm.gen_ai.server.time_per_output_token:PLAIN)%" gen_ai.server.time_to_first_token: "%FILTER_STATE(wasm.gen_ai.server.time_to_first_token:PLAIN)%"
Send a test request
Repeat the steps in Initiate a test to generate observability data for the gateway.
View the observability data
-
Get the name of the gateway workload.
export GATEWAY_DEPLOYMENT=$(kubectl -n envoy-gateway-system get deployment -l gateway.envoyproxy.io/owning-gateway-name=mock-gateway -o jsonpath='{.items[0].metadata.name}') echo $GATEWAY_DEPLOYMENT -
Forward the gateway's admin port to your local machine.
kubectl -n envoy-gateway-system port-forward deployments/$GATEWAY_DEPLOYMENT 19000:19000 -
Open a new terminal window and query the gateway metrics.
curl -s localhost:19000/stats/prometheus | grep gen_aiExpected output:
# TYPE gen_ai_client_operation_duration histogram gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="0.5"} 0 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="1"} 0 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="5"} 9 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="10"} 9 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="25"} 14 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="50"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="100"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="250"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="500"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="1000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="2500"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="5000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="10000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="30000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="60000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="300000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="600000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="1800000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="3600000"} 16 gen_ai_client_operation_duration_bucket{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000",le="+Inf"} 16 gen_ai_client_operation_duration_sum{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000"} 140.9499999999999886313162278384 gen_ai_client_operation_duration_count{gen_ai_operation_name="chat",gen_ai_system="example.com",gen_ai_request_model="mock",gen_ai_response_model="mock",gen_ai_error_type="",server_port="8000",server_address="10.3.0.9:8000"} 16 -
View the access logs.
kubectl -n envoy-gateway-system logs deployments/$GATEWAY_DEPLOYMENT | tail -1Expected output:
Defaulted container "envoy" out of: envoy, shutdown-manager { ":authority": "example.com", "bytes_received": 184, "bytes_sent": 355, "connection_termination_details": null, "downstream_local_address": "10.3.0.38:10080", "downstream_remote_address": "10.3.15.252:45492", "duration": 2, "gen_ai.completion.tokens": "76", "gen_ai.error.type": "", "gen_ai.operation.name": "chat", "gen_ai.prompt.tokens": "18", "gen_ai.request.model": "mock", "gen_ai.response.model": "mock", "gen_ai.server.time_per_output_token": "0", "gen_ai.server.time_to_first_token": "2", "gen_ai.system": "example.com", "method": "POST", "protocol": "HTTP/1.1", "requested_server_name": null, "response_code": 200, "response_code_details": "via_upstream", "response_flags": "-", "route_name": "httproute/default/mock-route/rule/0/match/0/*", "start_time": "2025-05-28T06:13:31.190Z", "upstream_cluster": "httproute/default/mock-route/rule/0/backend/0", "upstream_host": "10.3.0.9:8000", "upstream_local_address": "10.3.0.38:33370", "upstream_transport_failure_reason": null, "user-agent": "curl/8.8.0", "x-envoy-origin-path": "/v1/chat/completions", "x-envoy-upstream-service-time": null, "x-forwarded-for": "10.3.15.252", "x-request-id": "0e67d734-aca7-4c80-bda3-79641cd63e2c" }For metric descriptions and log field definitions, see OpenTelemetry (OTel) GenAI Semantic Conventions.
FAQ
Resolving the "413 Request Entity Too Large" error
Cause: When the observability plugin is enabled, the gateway buffers the entire request body to parse its content. If the request body exceeds the default buffer limit, the request fails and the gateway returns an HTTP 413 Request Entity Too Large error.
Solution: You can increase the buffer limit of the gateway by creating a ClientTrafficPolicy resource.
-
Create a file named
client-buffer-limit.yamlwith the following content. Replace${gateway-name}with the actual gateway name (theGatewayresource'smetadata.name). -
Run the following command to apply the configuration:
kubectl apply -f client-buffer-limit.yaml
gen-ai-telemetry plugin release history
|
Image tag |
Release time |
Description |
|
g2ad0869-aliyun |
May 2025 |
Supports metric monitoring and log enhancement for generative AI requests. |
|
g76f5a66-aliyun |
August 2025 |
Fixed the issue where the number of tokens for streaming requests was recorded inaccurately. |