Alibaba Cloud Service Mesh:Access an LLM service from a cluster using an ASM egress gateway

Prerequisites

• An ACK cluster is added to an ASM instance of version 1.22 or later.

• A sidecar injection policy is configured.

• An ingress gateway is created.

• The sleep sample application is deployed. For more information, see Create the sleep test Application.

• You have activated Alibaba Cloud Model Studio and obtained a valid API key. For more information, see Get an API Key.

Overview

Without an egress gateway, client requests are intercepted by the sidecar proxy, which then sends the requests to the LLM provider. Because the sidecar and the client are deployed in the same Pod, this method poses several security risks, such as API key leaks and unauthorized access. If you are concerned about these risks, we strongly recommend using an egress gateway to route requests.

Using an ASM egress gateway improves security. Because the egress gateway is deployed independently of the client, you can use the Role-Based Access Control (RBAC) mechanism of your ACK cluster to restrict who can manage it, improving the gateway's overall security. The egress gateway handles all security operations, such as dynamically adding an API key and enforcing authentication and authorization policies. This design lets you use the full range of ASM gateway security features.

The following diagram shows the request path used in this topic:

Step 1: Create an egress gateway and Gateway resource

1. Create an egress gateway, expose port 80, and enable Support two-way TLS authentication. For details, see Create an Egress Gateway.

2. Create a file named egress-gw.yaml with the following content.

   apiVersion: networking.istio.io/v1beta1
   kind: Gateway
   metadata:
     name: egress-gw
     namespace: istio-system
   spec:
     selector:
       istio: egressgateway
     servers:
       - hosts:
           - '*'
         port:
           name: http
           number: 80
           protocol: HTTPS
         tls:
           mode: ISTIO_MUTUAL

3. Use the kubeconfig file for your ASM instance to run the following command and create the Gateway resource.

   kubectl apply -f egress-gw.yaml

Step 2: Create an LLMProvider for the egress gateway

Applying the LLMProvider only to the egress gateway ensures the API key is stored solely in the gateway's memory, making it inaccessible to the client.

1. Create a file named dashscope-qwen.yaml with the following content.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMProvider
   metadata:  
     name: dashscope-qwen
     namespace: istio-system
   spec:
     workloadSelector:
       labels:
         istio: egressgateway
     host: dashscope.aliyuncs.com
     path: /compatible-mode/v1/chat/completions
     configs:
       defaultConfig:
         openAIConfig:
           model: qwen1.5-72b-chat  # The Qwen series of open-source large models
           stream: false
           apiKey: ${API_KEY}

2. Run the following command to create the LLMProvider.

   kubectl apply -f dashscope-qwen.yaml

Step 3: Create an LLMRoute

1. Create a file named dashscope-route.yaml with the following content to route traffic to the egress gateway.

   apiVersion: istio.alibabacloud.com/v1beta1
   kind: LLMRoute
   metadata:  
     name: dashscope-route
   spec:
     host: dashscope.aliyuncs.com
     gateways:
     - mesh
     - istio-system/egress-gw
     rules:
     - name: mesh-route # When the sidecar receives a request for dashscope.aliyuncs.com, it forwards the request to the egress gateway.
       matches:
       - gateways:
         - mesh
       backendRefs:
       - providerHost: istio-egressgateway.istio-system.svc.cluster.local
     - name: egress-gw-route # When the egress gateway receives a request for dashscope.aliyuncs.com, it forwards the request to the actual provider.
       matches:
       - gateways:
         - istio-system/egress-gw
       backendRefs:
       - providerHost: dashscope.aliyuncs.com

2. Run the following command to create the LLMRoute.

   kubectl apply -f dashscope-route.yaml

Step 4: Test the configuration

Use the kubeconfig file for your ACK cluster to run the following command to test the setup.

kubectl exec deployment/sleep -it -- curl --location 'http://dashscope.aliyuncs.com' \
--header 'Content-Type: application/json' \
--data '{
    "messages": [
        {"role": "user", "content": "Introduce yourself"}
    ]
}'

Expected output:

{"choices":[{"message":{"role":"assistant","content":"Hello! I am Qwen, a pre-trained language model developed by Alibaba Cloud. My purpose is to assist users in generating various types of text, such as articles, stories, poems, and answering questions by leveraging my extensive knowledge and understanding of context. Although I'm an AI, I don't have a physical body or personal experiences like human beings do, but I've been trained on a vast corpus of text data, which allows me to engage in conversations, provide information, or help with various tasks to the best of my abilities. So, feel free to ask me anything, and I'll do my best to provide helpful and informative responses!"},"finish_reason":"stop","index":0,"logprobs":null}],"object":"chat.completion","usage":{"prompt_tokens":12,"completion_tokens":130,"total_tokens":142},"created":1720680044,"system_fingerprint":null,"model":"qwen1.5-72b-chat","id":"chatcmpl-3608dcd5-e3ad-9ade-bc70-xxxxxxxxxxxxxx"}

Step 5: Configure an authorization policy

1. Create a file named authpolicy.yaml with the following content.

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: test
     namespace: istio-system
   spec:
     action: DENY
     rules:
     - from:
       - source:
           principals:
           - cluster.local/ns/default/sa/sleep
       to:
       - operation:
           hosts:
           - dashscope.aliyuncs.com
     selector:
       matchLabels:
         istio: egressgateway

2. Use the kubeconfig file for your ASM instance to run the following command to apply the authorization policy.

   kubectl apply -f authpolicy.yaml

3. Run the command from Step 4 again to test the policy.

   Expected output:

   RBAC: access denied

   The output shows that the request is denied.