Alibaba Cloud Service Mesh:Accelerate AI model serving with KServe on ASM and Fluid

How it works

This integration connects three components:

1. Fluid caches model files from an OSS bucket to local SSD storage across cluster nodes. A JindoRuntime custom resource (CR) manages the distributed cache, and a DataLoad CR prefetches data before any inference pod starts.

2. KServe on ASM deploys an InferenceService CR that references cached data through a persistent volume claim (PVC). The inference container loads the model from the Fluid-managed mount point instead of downloading it from OSS.

3. Service Mesh (ASM) routes traffic through an ingress gateway to the KServe inference service. Knative Serving handles autoscaling and scale-to-zero.

When a request arrives and Knative scales a pod from zero, the model is already cached locally. The pod starts and serves predictions without a remote download.

Prerequisites

Before you begin, make sure that you have:

• An ASM instance of version 1.17 or later with a Kubernetes cluster added to it. For more information, see Create an ASM instance and Add a cluster to an ASM instance

• One of the following Kubernetes clusters added to the ASM instance:

  • A Container Service for Kubernetes (ACK) cluster of version 1.22 or later. For GPU-based inference, the cluster must include GPU-accelerated nodes such as ecs.gn6i-c16g1.4xlarge. For more information, see Create an ACK managed cluster or Update an ACK cluster

  • An ACK Serverless cluster of version 1.18 or later with the CoreDNS component installed. For more information, see Create an ACK Serverless cluster and Manage system components

• Kubernetes API access to Istio resources enabled for the ASM instance. For more information, see Use the Kubernetes API of clusters on the data plane to access Istio resources

• An ASM ingress gateway named ingressgateway with ports 80 and 443 exposed. For more information, see Create an ingress gateway

• The Knative Serving component deployed in the cluster with the Knative on ASM feature enabled. For more information, see Use Knative on ASM to deploy a serverless application

• An active OSS subscription with at least one bucket created. For more information, see Activate OSS and Create buckets

Step 1: Enable KServe on ASM

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Ecosystem > KServe on ASM.

3. On the KServe on ASM page, configure the CertManager option and click Enable KServe on ASM. KServe depends on cert-manager for certificate lifecycle management. The CertManager component is automatically installed with KServe.

   • If CertManager is not installed in the cluster, turn on Automatically install the CertManager component in the cluster.

   • If CertManager is already installed in the cluster, turn off Automatically install the CertManager component in the cluster.

Step 2: Install Fluid and set up model caching

Step 3: Deploy the inference service

1. Create a file named oss-fluid-isvc.yaml with the InferenceService configuration that matches your cluster type.

   ACK cluster

      apiVersion: "serving.kserve.io/v1beta1"
      kind: "InferenceService"
      metadata:
        name: "fluid-bloom"
      spec:
        predictor:
          timeout: 600
          minReplicas: 0
          containers:
            - name: kserve-container
              image: registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu
              resources:
                limits:
                  cpu: "12"
                  memory: 48Gi
                  nvidia.com/gpu: 1  # Number of GPUs required. Remove this line if GPUs are not used.
                requests:
                  cpu: "12"
                  memory: 48Gi
              env:
                - name: STORAGE_URI
                  value: "pvc://oss-data/bloom-560m"
                - name: MODEL_NAME
                  value: "bloom"
                - name: GPU_ENABLED
                  value: "True"  # Set to "False" if GPUs are not used.

   ACK Serverless cluster

      apiVersion: "serving.kserve.io/v1beta1"
      kind: "InferenceService"
      metadata:
        name: "fluid-bloom"
        labels:
          alibabacloud.com/fluid-sidecar-target: "eci"
        annotations:
          k8s.aliyun.com/eci-use-specs: "ecs.gn6i-c16g1.4xlarge"  # ECS instance type
          knative.aliyun.com/reserve-instance-eci-use-specs: "ecs.gn6i-c16g1.4xlarge"  # ECS instance type
      spec:
        predictor:
          timeout: 600
          minReplicas: 0
          containers:
            - name: kserve-container
              image: registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu
              resources:
                limits:
                  cpu: "12"
                  memory: 48Gi
                requests:
                  cpu: "12"
                  memory: 48Gi
              env:
                - name: STORAGE_URI
                  value: "pvc://oss-data/bloom-560m"
                - name: MODEL_NAME
                  value: "bloom"
                - name: GPU_ENABLED
                  value: "True"  # Set to "False" if GPUs are not used.

   The following table describes the InferenceService parameters:

   Parameter	Value	Description
   image	registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu	Sample image with model loading and inference interfaces. To customize, see the KServe Fluid Docker samples
   STORAGE_URI	pvc://oss-data/bloom-560m	PVC-based storage URI pointing to the Fluid-managed cache
   MODEL_NAME	bloom	Model name used in the prediction API endpoint
   GPU_ENABLED	True	Set to False if GPUs are not used
   resources	12 CPU, 48 GiB memory	Resource allocation for the BLOOM-560m model. Adjust based on your model size and cluster capacity

2. Deploy the inference service:

      kubectl create -f oss-fluid-isvc.yaml -n kserve-fluid-demo

3. Verify the deployment:

      kubectl get inferenceservice -n kserve-fluid-demo

   Expected output:

      NAME          URL                                                READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION           AGE
      fluid-bloom   http://fluid-bloom.kserve-fluid-demo.example.com   True           100                              fluid-bloom-predictor-00001   2d

   READY shows True, which confirms the inference service is running.

Step 4: Send a prediction request

1. Get the ASM ingress gateway address.

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.

   3. In the Service address section of ingressgateway, copy the service address.

2. Send a test request. Replace <gateway-address> with the address from the previous step.

      curl -v \
        -H "Content-Type: application/json" \
        -H "Host: fluid-bloom.kserve-fluid-demo.example.com" \
        "http://<gateway-address>:80/v1/models/bloom:predict" \
        -d '{"prompt": "It was a dark and stormy night", "result_length": 50}'

   Expected output:

      *   Trying xxx.xx.xx.xx:80...
      * Connected to xxx.xx.xx.xx (xxx.xx.xx.xx) port 80 (#0)
      > POST /v1/models/bloom:predict HTTP/1.1
      > Host: fluid-bloom-predictor.kserve-fluid-demo.example.com
      > Content-Type: application/json
      >
      < HTTP/1.1 200 OK
      < content-type: application/json
      < server: istio-envoy
      <
      {
        "result": "It was a dark and stormy night, and the wind was blowing in the\ndirection of the west. The wind was blowing in the direction of the\nwest, and the wind was blowing in the direction of the west. The\nwind was"
      }

   A 200 OK response with generated text confirms that the inference service is working.

Clean up

To remove all resources created in this tutorial, run the following commands in order:

kubectl delete inferenceservice fluid-bloom -n kserve-fluid-demo
kubectl delete dataload oss-dataload -n kserve-fluid-demo
kubectl delete dataset oss-data -n kserve-fluid-demo
kubectl delete jindoruntime oss-data -n kserve-fluid-demo
kubectl delete secret access-key -n kserve-fluid-demo
kubectl delete ns kserve-fluid-demo

What's next

• Overview of Fluid

• Overview of Knative on ASM

• Use Knative on ASM to deploy a serverless application

• Integrate the cloud-native inference service KServe with ASM