Most major large language model (LLM) providers offer services over HTTP. Service Mesh (ASM) optimizes LLM request handling over HTTP, with built-in support for multiple major LLM provider protocols. You can use ASM to route LLM traffic across providers and monitor request-level metrics such as token usage.
Feature overview
Traffic routing
To register an external HTTP service in a service mesh cluster, you configure a ServiceEntry and then define routing rules with a VirtualService. You can then call this service through a gateway or an application pod. Without registration, the traffic management and observability features of the service mesh are unavailable.
However, a native ServiceEntry handles only regular TCP and HTTP traffic. LLM requests include advanced parameters that extend HTTP, which a standard ServiceEntry cannot support. ASM introduces two new resources to address this:
-
LLMProvider: Analogous to a ServiceEntry for HTTP. Use an LLMProvider to register an external LLM service provider with the cluster and configure its host, API key, and model parameters.
-
LLMRoute: Analogous to a VirtualService for HTTP. Use an LLMRoute to define traffic rules that distribute requests to specific LLMProviders based on weights or matching conditions.
Based on the LLMRoute and LLMProvider configurations, ASM dynamically selects a routing destination, applies preconfigured request parameters, and forwards the request to the target provider. This lets you switch provider configurations, select models based on request characteristics, and perform canary traffic shifting between providers, significantly reducing the complexity of integrating LLMs into the cluster. The following two scenarios illustrate LLM traffic management with LLMRoute and LLMProvider.
Configure an LLMRoute to use different models for different user types
Alibaba Cloud Model Studio provides two models: qwen-1.8b-chat and qwen-turbo. You can configure an LLMRoute to route regular users to the default qwen-1.8b-chat model and subscribed users to the more powerful qwen-turbo model. Subscribed user requests contain a special header that identifies their status.
Configure an LLMProvider and an LLMRoute to distribute traffic by weight
This scenario combines Alibaba Cloud Model Studio and Moonshot language model services. You can configure an LLMRoute and an LLMProvider to distribute traffic across providers by weight.
demo-llm-server is a regular service in the cluster and does not correspond to any endpoint.
Traffic observability
Beyond LLM request routing, ASM provides enhanced observability for LLM scenarios. Accurate and clear observable data lets O&M staff and developers monitor service health and respond to issues promptly.
Service mesh observability includes three main components:
-
Access logs
-
Monitoring metrics
-
Tracing Analysis
LLM requests use HTTP, so they are directly compatible with the existing Tracing Analysis feature. However, the default access log and monitoring metrics features lack LLM-specific details. For example, access logs cannot include the model used in a request, and monitoring metrics reflect only standard HTTP information. ASM enhances these features in two ways:
-
Access logs: Use the custom access log format feature to include LLM-specific information in access logs.
-
Monitoring metrics:
-
ASM adds two monitoring metrics that show the number of input tokens (prompt tokens) and output tokens (completion tokens) per request.
-
LLM-specific information is added as a metric dimension that you can reference in standard Istio metrics.
-
Scenario overview
Integrating LLMs with ASM enables canary releases, weighted routing, and observability features. This decouples applications from LLM providers and improves the robustness and maintainability of the call chain. The following scenarios cover LLM traffic routing and observability configuration.