Traffic observability: Manage LLM traffic with ASM - Alibaba Cloud Service Mesh

In addition to the LLM traffic routing capabilities described in the previous topic, ASM enhances its observability for LLM scenarios. This topic explains how to use the ASM access log and monitoring metrics to observe LLM request information.

Important

To demonstrate the full range of features, this topic assumes that you have completed all the steps in Traffic routing: Use ASM to efficiently manage LLM traffic. If you have completed only Step 1 and Step 2 of that topic, you can still send test requests with the commands from Step 2. The commands used to view observability data are the same as those in this topic.

Step 1: Observe LLM requests with access logs

Configure the access log

ASM enhances logging for LLM requests. To view this information in the access log, simply update the custom access log format. For more information, see custom data plane access log.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
In the global Log Settings, add the following three fields.

The text content is as follows:
```
request_model				FILTER_STATE(wasm.asm.llmproxy.request_model:PLAIN)
request_prompt_tokens			FILTER_STATE(wasm.asm.llmproxy.request_prompt_tokens:PLAIN)
request_completion_tokens		FILTER_STATE(wasm.asm.llmproxy.request_completion_tokens:PLAIN)
```
The fields are defined as follows:
- request_model: The actual model used for the current LLM request, such as qwen-turbo or qwen1.5-72b-chat.
- request_prompt_tokens: The number of input tokens for the request.
- request_completion_tokens: The number of output tokens for the request.
Most LLM service providers charge based on token usage. This data allows you to track token-consuming requests and identify the models used.

Verification

Using the kubeconfig for your ACK cluster, run the following two commands separately.

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--header 'user-type: subscriber' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

Run the following command to view the access log.

kubectl logs deployments/sleep -c istio-proxy | tail -2

Expected output:

{"bytes_received":"85","bytes_sent":"617","downstream_local_address":"47.93.xxx.xx:80","downstream_remote_address":"192.168.34.235:39066","duration":"7640","istio_policy_status":"-","method":"POST","path":"/compatible-mode/v1/chat/completions","protocol":"HTTP/1.1","request_id":"d0e17f66-f300-411a-8c32-xxxxxxxxxxxxx","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"-","start_time":"2024-07-12T03:20:03.993Z","trace_id":"-","upstream_cluster":"outbound|80||dashscope.aliyuncs.com","upstream_host":"47.93.xxx.xx:443","upstream_local_address":"192.168.34.235:38476","upstream_service_time":"7639","upstream_response_time":"7639","upstream_transport_failure_reason":"-","user_agent":"curl/8.8.0","x_forwarded_for":"-","authority_for":"dashscope.aliyuncs.com","request_model":"qwen1.5-72b-chat","request_prompt_tokens":"3","request_completion_tokens":"55"}
{"bytes_received":"85","bytes_sent":"809","downstream_local_address":"47.93.xxx.xx:80","downstream_remote_address":"192.168.34.235:41090","duration":"2759","istio_policy_status":"-","method":"POST","path":"/compatible-mode/v1/chat/completions","protocol":"HTTP/1.1","request_id":"d89faada-6af3-4ac3-b4fd-xxxxxxxxxxxxx","requested_server_name":"-","response_code":"200","response_flags":"-","route_name":"vip-route","start_time":"2024-07-12T03:20:30.854Z","trace_id":"-","upstream_cluster":"outbound|80||dashscope.aliyuncs.com","upstream_host":"47.93.xxx.xx:443","upstream_local_address":"192.168.34.235:38476","upstream_service_time":"2759","upstream_response_time":"2759","upstream_transport_failure_reason":"-","user_agent":"curl/8.8.0","x_forwarded_for":"-","authority_for":"dashscope.aliyuncs.com","request_model":"qwen-turbo","request_prompt_tokens":"11","request_completion_tokens":"90"}

After formatting, the log content is as follows:

{
    "duration": "7640",
    "response_code": "200",
    "authority_for": "dashscope.aliyuncs.com",  --Actual LLM provider accessed
    "request_model": "qwen1.5-72b-chat",    	--Model used by the current request
    "request_prompt_tokens": "3",		--Number of input tokens for the current request
    "request_completion_tokens": "55"		--Number of output tokens for the current request
}

{
  "duration": "2759",
  "response_code": "200",
  "authority_for": "dashscope.aliyuncs.com",  --Actual LLM provider accessed
  "request_model": "qwen-turbo",    	      --Model used by the current request
  "request_prompt_tokens": "11",	      --Number of input tokens for the current request
  "request_completion_tokens": "90"	      --Number of output tokens for the current request 
}

ASM is integrated with SLS. By collecting and storing these access logs, you can observe request-level LLM calls, define custom alert rules, and build insightful dashboards. For more information, see Enable data plane log collection.

Step 2: Add metrics for token consumption

While the access log provides fine-grained information, monitoring metrics offer a higher-level view. The ASM mesh proxy exports token consumption at the workload level as monitoring metrics, allowing you to observe a workload's token usage in real time.

ASM adds two new metrics:

asm_llm_proxy_prompt_tokens: The number of input tokens.
asm_llm_proxy_completion_tokens: The number of output tokens.

By default, these two metrics have the following dimensions:

llmproxy_source_workload: The name of the workload that sends the request.
llmproxy_source_workload_namespace: The namespace where the request source is located.
llmproxy_destination_service: The destination provider.
llmproxy_model: The model for the current request.

Modify workload configuration to output new metrics

This step uses the sleep deployment in the default namespace as an example.

Use the kubeconfig file of your ACK cluster to create a file named asm-llm-proxy-bootstrap-config.yaml.

apiVersion: v1
kind: ConfigMap
metadata:
  name: asm-llm-proxy-bootstrap-config
data:
  custom_bootstrap.json: |
    "stats_config": {
      "stats_tags":[
        {
        "tag_name": "llmproxy_source_workload",
        "regex": "(\\|llmproxy_source_workload=([^|]*))"
        },
        {
          "tag_name": "llmproxy_source_workload_namespace",
          "regex": "(\\|llmproxy_source_workload_namespace=([^|]*))"
        },
        {
          "tag_name": "llmproxy_destination_service",
          "regex": "(\\|llmproxy_destination_service=([^|]*))"
        },
        {
          "tag_name": "llmproxy_model",
          "regex": "(\\|llmproxy_model=([^|]*))"
        }
      ]
    }

Run the following command to create a ConfigMap named asm-llm-proxy-bootstrap-config.
```
kubectl apply -f asm-llm-proxy-bootstrap-config.yaml
```

Run the following command to modify the sleep deployment and add an annotation to the pod.

kubectl patch deployment sleep -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/bootstrapOverride":"asm-llm-proxy-bootstrap-config"}}}}}'

Verification

Run the following two commands separately to send test requests.

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--header 'user-type: subscriber' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

Run the following command to view the Prometheus metrics exported by the sidecar of the sleep application.

kubectl exec deployments/sleep -it -c istio-proxy -- curl localhost:15090/stats/prometheus | grep llmproxy

Expected output:

asm_llm_proxy_completion_tokens{llmproxy_source_workload="sleep",llmproxy_source_workload_namespace="default",llmproxy_destination_service="dashscope.aliyuncs.com",llmproxy_model="qwen1.5-72b-chat"} 72
asm_llm_proxy_completion_tokens{llmproxy_source_workload="sleep",llmproxy_source_workload_namespace="default",llmproxy_destination_service="dashscope.aliyuncs.com",llmproxy_model="qwen-turbo"} 85
asm_llm_proxy_prompt_tokens{llmproxy_source_workload="sleep",llmproxy_source_workload_namespace="default",llmproxy_destination_service="dashscope.aliyuncs.com",llmproxy_model="qwen1.5-72b-chat"} 3
asm_llm_proxy_prompt_tokens{llmproxy_source_workload="sleep",llmproxy_source_workload_namespace="default",llmproxy_destination_service="dashscope.aliyuncs.com",llmproxy_model="qwen-turbo"} 11

The output confirms that the sidecar exports the metrics, each including four default dimensions.

ASM is integrated with ARMS. For more detailed analysis and visualization, configure collection rules to send these metrics to Managed Service for Prometheus. For more information, see Collect monitoring metrics to Managed Service for Prometheus.

Step 3: Add LLM dimensions to native metrics

The service mesh provides many default metrics that display detailed information for HTTP and TCP protocols. These metrics offer a wealth of dimensions, and ASM provides powerful, built-in Prometheus dashboards based on them.

However, these metrics do not contain information about LLM requests. To address this, ASM allows you to add LLM request information to existing metrics by customizing their dimensions.

Configure a custom dimension: model

This section demonstrates how to add the model dimension to the REQUEST_COUNT metric.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Observability Management Center > Observability Settings.
Click Edit Dimension for Edit dimension, select the Custom Dimension tab, and click Custom Dimensions. Set Dimension Name to model and Value to filter_state["wasm.asm.llmproxy.request_model"].

Verification

Run the following two commands separately to send test requests.

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--header 'user-type: subscriber' \
--data '{
    "messages": [
        {"role": "user", "content": "Please introduce yourself"}
    ]
}'

Run the following command to view the Prometheus metrics exported by the sidecar of the sleep application.

kubectl exec deployments/sleep -it -c istio-proxy -- curl localhost:15090/stats/prometheus | grep istio_requests_total

Expected output:

istio_requests_total{reporter="source",source_workload="sleep",source_canonical_service="sleep",source_canonical_revision="latest",source_workload_namespace="default",source_principal="unknown",source_app="sleep",source_version="",source_cluster="cce8d2c1d1e8d4abc8d5c180d160669cc",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="dashscope.aliyuncs.com",destination_canonical_service="unknown",destination_canonical_revision="latest",destination_service_name="dashscope.aliyuncs.com",destination_service_namespace="unknown",destination_cluster="unknown",request_protocol="http",response_code="200",grpc_response_status="",response_flags="-",connection_security_policy="unknown",model="qwen1.5-72b-chat"} 1
istio_requests_total{reporter="source",source_workload="sleep",source_canonical_service="sleep",source_canonical_revision="latest",source_workload_namespace="default",source_principal="unknown",source_app="sleep",source_version="",source_cluster="cce8d2c1d1e8d4abc8d5c180d160669cc",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="dashscope.aliyuncs.com",destination_canonical_service="unknown",destination_canonical_revision="latest",destination_service_name="dashscope.aliyuncs.com",destination_service_namespace="unknown",destination_cluster="unknown",request_protocol="http",response_code="200",grpc_response_status="",response_flags="-",connection_security_policy="unknown",model="qwen-turbo"} 1

The output confirms that the model dimension has been added to the istio_requests_total metric.

With these monitoring metrics, you can configure analysis rules in ARMS for deeper insights. For example:

Request success rate for a specific model.
Average response latency for a specific model or provider.

Conclusion

This topic builds on Traffic routing: Use ASM to efficiently manage LLM traffic and describes how to perform both fine-grained and high-level observation of LLM traffic by using ASM. You can enable multi-dimensional observability features with only minor modifications to your cluster configuration. ASM is continuously improving its observability capabilities for LLM traffic to provide more detailed and flexible solutions.