Alibaba Cloud Service Mesh:Manage LLM traffic with ASM

Etapa 1: Implante a aplicação de teste sleep

O pod sleep atua como cliente para enviar requisições de teste aos provedores de LLM através do mesh.

1. Salve o conteúdo a seguir como sleep.yaml:

   YAML content

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: sleep
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: sleep
     labels:
       app: sleep
       service: sleep
   spec:
     ports:
     - port: 80
       name: http
     selector:
       app: sleep
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: sleep
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sleep
     template:
       metadata:
         labels:
           app: sleep
       spec:
         terminationGracePeriodSeconds: 0
         serviceAccountName: sleep
         containers:
         - name: sleep
           image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
           command: ["/bin/sleep", "infinity"]
           imagePullPolicy: IfNotPresent
           volumeMounts:
           - mountPath: /etc/sleep/tls
             name: secret-volume
         volumes:
         - name: secret-volume
           secret:
             secretName: sleep-secret
             optional: true
   ---

2. Aplique o manifesto ao seu cluster ACK:

   kubectl apply -f sleep.yaml

Etapa 2: Configure o provedor Model Studio

Crie um recurso LLMProvider para informar ao ASM como acessar o Alibaba Cloud Model Studio (DashScope).

1. Salve o conteúdo a seguir como LLMProvider.yaml. Substitua <your-dashscope-api-key> pela sua chave de API do Model Studio.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: dashscope-qwen
   spec:
     host: dashscope.aliyuncs.com
     path: /compatible-mode/v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: qwen1.5-72b-chat  # Qwen open-source series model
           apiKey: <your-dashscope-api-key>

   Para obter a lista completa de modelos open-source Qwen disponíveis, consulte Geração de texto - Modelos open-source Qwen.

2. Aplique o manifesto à sua instância do ASM:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider.yaml

3. Verifique a configuração enviando uma requisição de teste a partir do pod sleep:

   kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
     --header 'Content-Type: application/json' \
     --data '{
       "messages": [
         {"role": "user", "content": "Please introduce yourself."}
       ]
     }'

   Uma resposta bem-sucedida tem o seguinte formato:

   {
     "choices": [
       {
         "message": {
           "role": "assistant",
           "content": "Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud..."
         },
         "finish_reason": "stop",
         "index": 0
       }
     ],
     "model": "qwen1.5-72b-chat",
     "usage": {
       "prompt_tokens": 12,
       "completion_tokens": 130,
       "total_tokens": 142
     }
   }

   A requisição usa http://dashscope.aliyuncs.com simples, sem especificar caminho, modelo ou chave de API. O sidecar do ASM preenche esses campos a partir da configuração do LLMProvider, faz o upgrade para HTTPS e encaminha a requisição ao DashScope. Como o Model Studio é compatível com o protocolo OpenAI, a resposta segue o formato padrão de conclusão de chat.

Crie a regra de roteamento

1. Salve o conteúdo a seguir como LLMRoute.yaml:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMRoute
   metadata:
     name: dashscope-route
   spec:
     host: dashscope.aliyuncs.com  # Must match the LLMProvider host
     rules:
     - name: vip-route
       matches:
       - headers:
           user-type:
             exact: subscriber  # Match requests with this header value
       backendRefs:
       - providerHost: dashscope.aliyuncs.com
     - backendRefs:
       - providerHost: dashscope.aliyuncs.com

   A primeira regra corresponde a requisições que incluem o cabeçalho user-type: subscriber e as roteia pela regra vip-route. A segunda regra atua como captura padrão para todas as demais requisições.

2. Aplique a regra de roteamento à sua instância do ASM:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMRoute.yaml

Atribua um modelo a cada rota

Atualize o LLMProvider para especificar modelos diferentes para a rota padrão e para a vip-route:

1. Atualize o arquivo LLMProvider.yaml com o seguinte conteúdo:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: dashscope-qwen
   spec:
     host: dashscope.aliyuncs.com
     path: /compatible-mode/v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: qwen1.5-72b-chat  # Default: open-source model
           apiKey: <your-dashscope-api-key>
       routeSpecificConfigs:
         vip-route:                  # Override for subscriber requests
           openAIConfig:
             model: qwen-turbo       # Subscribers use qwen-turbo
             apiKey: <your-dashscope-api-key>

2. Aplique a atualização:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider.yaml

Teste o roteamento

Envie duas requisições: uma sem o cabeçalho (rota padrão) e outra com o cabeçalho user-type: subscriber (rota VIP).

# Default route: uses qwen1.5-72b-chat
kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

# Subscriber route: uses qwen-turbo
kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
  --header 'Content-Type: application/json' \
  --header 'user-type: subscriber' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

Saída esperada:

{"choices":[{"message":{"role":"assistant","content":"Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud. My purpose is to assist users in generating various types of text, such as articles, stories, poems, and answering questions by leveraging my extensive knowledge and understanding of context. Although I'm an AI, I don't have a physical body or personal experiences like human beings do, but I've been trained on a vast corpus of text data, which allows me to engage in conversations, provide information, or help with various tasks to the best of my abilities. So, feel free to ask me anything, and I'll do my best to provide helpful and informative responses!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":12,"completion_tokens":130,"total_tokens":142},"created":1720680044,"system_fingerprint":null,"model":"qwen1.5-72b-chat","id":"chatcmpl-1c33b950-3220-9bfe-9066-xxxxxxxxxxxx"}

{"choices":[{"message":{"role":"assistant","content":"Hello, I'm Qwen, a large language model from Alibaba Cloud. As an AI assistant, my goal is to help users get accurate and useful information, and to solve their problems and confusions. I can provide knowledge in various fields, engage in conversation, and even create text. Please note that all the content I provide is based on the data I was trained on and may not include the latest events or personal information. If you have any questions, feel free to ask me at any time!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":11,"completion_tokens":85,"total_tokens":96},"created":1720683416,"system_fingerprint":null,"model":"qwen-turbo","id":"chatcmpl-9cbc7c56-06e9-9639-a50d-xxxxxxxxxxxx"}

Verifique o campo model em cada resposta. A requisição padrão retorna "model": "qwen1.5-72b-chat", enquanto a requisição de assinante retorna "model": "qwen-turbo".

Etapa 1: Configure o provedor Moonshot

1. Salve o conteúdo a seguir como LLMProvider-moonshot.yaml. Substitua <your-moonshot-api-key> pela sua chave de API da Moonshot AI.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:
     name: moonshot
   spec:
     host: api.moonshot.cn  # Must be unique across all LLMProviders
     path: /v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: moonshot-v1-8k
           stream: false
           apiKey: <your-moonshot-api-key>

2. Aplique o manifesto:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMProvider-moonshot.yaml

Etapa 2: Crie um serviço virtual de LLM

Crie um Serviço do Kubernetes como ponto de entrada único para requisições de LLM. Este serviço não possui pods de backend; o sidecar do ASM roteia todas as requisições para os provedores de LLM definidos no LLMRoute.

1. Salve o conteúdo a seguir como demo-llm-server.yaml:

   apiVersion: v1
   kind: Service
   metadata:
     name: demo-llm-server
     namespace: default
   spec:
     ports:
     - name: http
       port: 80
       protocol: TCP
       targetPort: 80
     selector:
       app: none
     type: ClusterIP

2. Aplique o manifesto:

   kubectl apply -f demo-llm-server.yaml

Etapa 3: Configure o roteamento ponderado

Crie um LLMRoute para distribuir o tráfego uniformemente entre DashScope e Moonshot:

1. Salve o conteúdo a seguir como LLMRoute.yaml:

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMRoute
   metadata:
     name: demo-llm-server
     namespace: default
   spec:
     host: demo-llm-server
     rules:
     - name: migrate-rule
       backendRefs:
       - providerHost: dashscope.aliyuncs.com
         weight: 50
       - providerHost: api.moonshot.cn
         weight: 50

   Ajuste os valores de weight para controlar a divisão do tráfego. Os pesos são relativos: 50/50 divide igualmente, enquanto 80/20 envia 80% para o DashScope e 20% para a Moonshot.

2. Aplique a regra de roteamento:

   kubectl --kubeconfig=${PATH_TO_ASM_KUBECONFIG} apply -f LLMRoute.yaml

Teste o roteamento ponderado

Envie várias requisições para o serviço virtual demo-llm-server e observe as respostas:

kubectl exec deployment/sleep -it -- curl --location 'http://demo-llm-server' \
  --header 'Content-Type: application/json' \
  --data '{
    "messages": [
      {"role": "user", "content": "Please introduce yourself."}
    ]
  }'

Execute o comando várias vezes. Algumas respostas vêm da Moonshot (identificadas por "model": "moonshot-v1-8k" e pelo nome do assistente Kimi), enquanto outras vêm do DashScope (identificadas por "model": "qwen1.5-72b-chat" e pelo nome do assistente Qwen).

Saída esperada:

{"id":"cmpl-cafd47b181204cdbb4a4xxxxxxxxxxxx","object":"chat.completion","created":1720687132,"model":"moonshot-v1-8k","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! I am an AI language model named Kimi. My main function is to help people generate human-like text. I can write articles, answer questions, provide advice, and more. I am trained on a massive amount of text data, so I can generate a wide variety of text. My goal is to help people communicate more effectively and solve problems."},"finish_reason":"stop"}],"usage":{"prompt_tokens":11,"completion_tokens":59,"total_tokens":70}}

{"choices":[{"message":{"role":"assistant","content":"Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud. My purpose is to assist users in generating various types of text, such as articles, stories, poems, and answering questions by leveraging my extensive knowledge and understanding of context. Although I'm an AI, I don't have a physical body or personal experiences like human beings do, but I've been trained on a vast corpus of text data, which allows me to engage in conversations, provide information, or help with various tasks to the best of my abilities. So, feel free to ask me anything, and I'll do my best to provide helpful and informative responses!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":12,"completion_tokens":130,"total_tokens":142},"created":1720687164,"system_fingerprint":null,"model":"qwen1.5-72b-chat","id":"chatcmpl-2443772b-4e41-9ea8-9bed-xxxxxxxxxxxx"}

A saída mostra que as requisições são distribuídas uniformemente entre a Moonshot e o Alibaba Cloud Model Studio.

Recursos do Kubernetes

Placeholders

Substitua os seguintes placeholders pelos valores reais antes de aplicar os manifestos YAML: