Alibaba Cloud Service Mesh:Manage LLM traffic with ASM

Paso 1: Desplegar la aplicación de prueba sleep

El pod sleep funciona como cliente para enviar solicitudes de prueba a los proveedores de LLM a través de la malla.

1. Guarde el siguiente contenido como sleep.yaml:

   Contenido YAML

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
       service: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         terminationGracePeriodSeconds: 0
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
           command: ["/bin/sleep", "infinity"]
           imagePullPolicy: IfNotPresent
           volumeMounts:
           - mountPath: /etc/sleep/tls
             name: secret-volume
         volumes:
         - name: secret-volume
           secret:
             secretName: sleep-secret
             optional: true
   ---

2. Aplique el manifiesto a su clúster de ACK:

   kubectl apply -f sleep.yaml

Paso 2: Configurar el proveedor de Model Studio

Cree un recurso LLMProvider que indique a ASM cómo conectarse a Alibaba Cloud Model Studio (DashScope).

1. Guarde el siguiente contenido como LLMProvider.yaml. Reemplace <your-dashscope-api-key> con su clave de API de Model Studio.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: dashscope-qwen
   spec:
     host: dashscope.aliyuncs.com
     path: /compatible-mode/v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: qwen1.5-72b-chat  # Qwen open-source series model
           apiKey: <your-dashscope-api-key>

   Para consultar la lista completa de modelos de código abierto Qwen disponibles, véase Text generation - Qwen open-source models.

2. Aplique el manifiesto a su instancia de ASM:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider.yaml

3. Compruebe la configuración mediante el envío de una solicitud de prueba desde el pod sleep:

   kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
     --header 'Content-Type: application/json' \
     --data '{
       "messages": [
         {"role": "user", "content": "Please introduce yourself."}
       ]
     }'

   Una respuesta exitosa tiene el siguiente aspecto:

   {
     "choices": [
       {
         "message": {
           "role": "assistant",
           "content": "Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud..."
         },
         "finish_reason": "stop",
         "index": 0
       }
     ],
     "model": "qwen1.5-72b-chat",
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 130,
       "total_tokens": 142
     }
   }

   La solicitud se dirige a http://dashscope.aliyuncs.com sin especificar ruta, modelo ni clave de API. El sidecar de ASM completa estos campos desde la configuración de LLMProvider, actualiza la conexión a HTTPS y reenvía la solicitud a DashScope. Dado que Model Studio es compatible con el protocolo de OpenAI, la respuesta sigue el formato estándar de finalización de chat.

Creación de la regla de enrutamiento

1. Guarde el siguiente contenido como LLMRoute.yaml:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMRoute
   metadata:
     name: dashscope-route
   spec:
     host: dashscope.aliyuncs.com  # Must match the LLMProvider host
     rules:
     - name: vip-route
       matches:
       - headers:
           user-type:
             exact: subscriber  # Match requests with this header value
       backendRefs:
       - providerHost: dashscope.aliyuncs.com
     - backendRefs:
       - providerHost: dashscope.aliyuncs.com

   La primera regla coincide con las solicitudes que incluyen un encabezado user-type: subscriber y las enruta mediante la regla de enrutamiento vip-route. La segunda regla actúa como captura predeterminada para todas las demás solicitudes.

2. Aplique la regla de enrutamiento a su instancia de ASM:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMRoute.yaml

Asignación de un modelo a cada ruta

Actualice LLMProvider para especificar distintos modelos para la ruta predeterminada y para vip-route:

1. Actualice LLMProvider.yaml con el siguiente contenido:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: dashscope-qwen
   spec:
     host: dashscope.aliyuncs.com
     path: /compatible-mode/v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: qwen1.5-72b-chat  # Default: open-source model
           apiKey: <your-dashscope-api-key>
       routeSpecificConfigs:
         vip-route:                  # Override for subscriber requests
           openAIConfig:
             model: qwen-turbo       # Subscribers use qwen-turbo
             apiKey: <your-dashscope-api-key>

2. Aplique la actualización:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider.yaml

Prueba del enrutamiento

Envíe dos solicitudes: una sin el encabezado (ruta predeterminada) y otra con el encabezado user-type: subscriber (ruta VIP):

# Default route: uses qwen1.5-72b-chat
kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

# Subscriber route: uses qwen-turbo
kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
  --header 'Content-Type: application/json' \
  --header 'user-type: subscriber' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

Salida esperada:

{"choices":[{"message":{"role":"assistant","content":"Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud. My purpose is to assist users in generating various types of text, such as articles, stories, poems, and answering questions by leveraging my extensive knowledge and understanding of context. Although I'm an AI, I don't have a physical body or personal experiences like human beings do, but I've been trained on a vast corpus of text data, which allows me to engage in conversations, provide information, or help with various tasks to the best of my abilities. So, feel free to ask me anything, and I'll do my best to provide helpful and informative responses!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":12,"completion_tokens":130,"total_tokens":142},"created":1720680044,"system_fingerprint":null,"model":"qwen1.5-72b-chat","id":"chatcmpl-1c33b950-3220-9bfe-9066-xxxxxxxxxxxx"}

{"choices":[{"message":{"role":"assistant","content":"Hello, I'm Qwen, a large language model from Alibaba Cloud. As an AI assistant, my goal is to help users get accurate and useful information, and to solve their problems and confusions. I can provide knowledge in various fields, engage in conversation, and even create text. Please note that all the content I provide is based on the data I was trained on and may not include the latest events or personal information. If you have any questions, feel free to ask me at any time!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":11,"completion_tokens":85,"total_tokens":96},"created":1720683416,"system_fingerprint":null,"model":"qwen-turbo","id":"chatcmpl-9cbc7c56-06e9-9639-a50d-xxxxxxxxxxxx"}

Revise el campo model en cada respuesta. La solicitud predeterminada devuelve "model": "qwen1.5-72b-chat", mientras que la solicitud de suscriptor devuelve "model": "qwen-turbo".

Paso 1: Configurar el proveedor de Moonshot

1. Guarde el siguiente contenido como LLMProvider-moonshot.yaml. Reemplace <your-moonshot-api-key> con su clave de API de Moonshot AI.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: moonshot
   spec:
     host: api.moonshot.cn  # Must be unique across all LLMProviders
     path: /v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: moonshot-v1-8k
           stream: false
           apiKey: <your-moonshot-api-key>

2. Aplique el manifiesto:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider-moonshot.yaml

Paso 2: Crear un servicio virtual de LLM

Cree un Service de Kubernetes como punto de entrada único para las solicitudes de LLM. Este servicio no tiene pods de respaldo; el sidecar de ASM enruta todas las solicitudes a los proveedores de LLM definidos en LLMRoute.

1. Guarde el siguiente contenido como demo-llm-server.yaml:

   apiVersion: v1
   kind: Service
   metadata:
     name: demo-llm-server
     namespace: default
   spec:
     ports:
     - name: http
       port: 80
       protocol: TCP
       targetPort: 80
     selector:
       app: none
     type: ClusterIP

2. Aplique el manifiesto:

   kubectl apply -f demo-llm-server.yaml

Paso 3: Configurar el enrutamiento ponderado

Cree un recurso LLMRoute que distribuya el tráfico equitativamente entre DashScope y Moonshot:

1. Guarde el siguiente contenido como LLMRoute.yaml:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMRoute
   metadata:
     name: demo-llm-server
     namespace: default
   spec:
     host: demo-llm-server
     rules:
     - name: migrate-rule
       backendRefs:
       - providerHost: dashscope.aliyuncs.com
         weight: 50
       - providerHost: api.moonshot.cn
         weight: 50

   Ajuste los valores de weight para controlar la división del tráfico. Los pesos son relativos: 50/50 divide el tráfico equitativamente, mientras que 80/20 envía el 80 % a DashScope y el 20 % a Moonshot.

2. Aplique la regla de enrutamiento:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMRoute.yaml

Prueba del enrutamiento ponderado

Envíe varias solicitudes al servicio virtual demo-llm-server y observe las respuestas:

kubectl exec deployment/sleep -it -- curl --location 'http://demo-llm-server' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

Ejecute el comando varias veces. Algunas respuestas provienen de Moonshot (identificadas por "model": "moonshot-v1-8k" y el nombre del asistente Kimi), mientras que otras provienen de DashScope (identificadas por "model": "qwen1.5-72b-chat" y el nombre del asistente Qwen).

Salida esperada:

{"id":"cmpl-cafd47b181204cdbb4a4xxxxxxxxxxxx","object":"chat.completion","created":1720687132,"model":"moonshot-v1-8k","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! I am an AI language model named Kimi. My main function is to help people generate human-like text. I can write articles, answer questions, provide advice, and more. I am trained on a massive amount of text data, so I can generate a wide variety of text. My goal is to help people communicate more effectively and solve problems."},"finish_reason":"stop"}],"usage":{"prompt_tokens":11,"completion_tokens":59,"total_tokens":70}}

{"choices":[{"message":{"role":"assistant","content":"Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud. My purpose is to assist users in generating various types of text, such as articles, stories, poems, and answering questions by leveraging my extensive knowledge and understanding of context. Although I'm an AI, I don't have a physical body or personal experiences like human beings do, but I've been trained on a vast corpus of text data, which allows me to engage in conversations, provide information, or help with various tasks to the best of my abilities. So, feel free to ask me anything, and I'll do my best to provide helpful and informative responses!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":12,"completion_tokens":130,"total_tokens":142},"created":1720687164,"system_fingerprint":null,"model":"qwen1.5-72b-chat","id":"chatcmpl-2443772b-4e41-9ea8-9bed-xxxxxxxxxxxx"}

La salida muestra que las solicitudes se distribuyen equitativamente entre Moonshot y Alibaba Cloud Model Studio.

Recursos de Kubernetes

Marcadores de posición

Reemplace los siguientes marcadores de posición con sus valores reales antes de aplicar los manifiestos YAML: