Description of LLMRoute CRD - Alibaba Cloud Service Mesh

LLMRoute is a customized Kubernetes CRD provided by Alibaba Cloud Service Mesh (ASM) designed to provide declarative configuration capabilities for routing rules related to Large Language Models (LLM). This CRD supports traffic matching based on conditions such as request headers and resource tags, and can flexibly route traffic to specified external LLM services or in-cluster inference pools. This topic details the fields of the LLMRoute resource.

Sample configuration

The following is a sample configuration of LLMRoute. For a complete example, see Traffic routing: Use ASM to manage LLM traffic.

apiVersion: istio.alibabacloud.com/v1beta1
kind: LLMRoute
metadata:  
  name: dashscope-route
spec:
  host: dashscope.aliyuncs.com # Must be unique between different LLM providers
  rules:
  - name: vip-route
    matches:
    - headers:
        user-type:
          exact: subscriber  # Routing item dedicated to subscribers
    backendRefs:
    - providerHost: dashscope.aliyuncs.com
  - backendRefs:
    - providerHost: dashscope.aliyuncs.com

Configuration items

LLMRoute

Configuration item	Type	Description
host	`String`	The URL of the route destination host.
gateways	`[]String`	The gateway rules, same as `gateways` in virtual service.
rules	`[]LLMRule`	The list of routing rules.

LLMRule

LLMRule is used to define a single routing rule.

Configuration item	Type	Description
name	`String`	The rule name.
matches	`[]LLMRequestMatch`	The match conditions.
backendRefs	`[]LLMBackendRef`	The list of backend references.

LLMRequestMatch

LLMRequestMatch defines the conditions that requests must match.

Configuration item	Type	Description
Headers	`map[String]StringMatch`	The request headers that requests must match.
SourceLabels	`map[String]String`	The resource labels that requests must match.
Gateways	`[]String`	The gateway rules that requests must match.

LLMBackendRef

LLMBackendRef defines the information referenced by backend objects, including the following fields:

Configuration item	Type	Description
ProviderHost	`String`	The target Host.
Weight	`Int32`	The weight value.
BackendRef	`BackendObjectReference`	The backend object reference.

ProviderHost and BackendRef cannot exist simultaneously.

BackendObjectReference

BackendObjectReference defines the reference information of the backend object, including the following fields:

Configuration item	Type	Description
Group	`String`	The group to which the backend object belongs.
Kind	`String`	The type of the backend object.
Name	`String`	The name of the backend object.
Namespace	`String`	The namespace of the backend object.
Port	`Int32`	The port of the backend object.

Currently, only InferencePool can be referenced.