Accelerate AI serving with KServe, Fluid, and ASM - Alibaba Cloud Service Mesh

Prerequisites

A Kubernetes cluster has been added to an ASM instance. For more information, see Create an ASM instance and Add a cluster to an ASM instance.
Note
- The ASM instance is version 1.17 or later. For more information about how to upgrade an instance, see Upgrade an ASM instance.
- Kubernetes cluster requirements:
  - The ACK cluster is version 1.22 or later. For more information, see Create an ACK managed cluster or Upgrade an ACK cluster. If you use a graphics processing unit (GPU) to run AI services, the ACK cluster must contain GPU-accelerated nodes, such as ecs.gn6i-c16g1.4xlarge.
  - The ACK Serverless cluster is version 1.18 or later, and the CoreDNS component is installed. For more information, see Create an ACK Serverless cluster and Manage components.
Istio resource access through the Kubernetes API of the data plane cluster must be enabled for the ASM instance. For more information, see Use the Kubernetes API of a data plane cluster to access Istio resources.
An ingress gateway has been created for the cluster. This topic uses an ASM ingress gateway named ingressgateway that exposes ports 80 and 443. For more information, see Create an ingress gateway.
The Knative Serving component is deployed in the ACK or ACK Serverless cluster and Knative on ASM is enabled. For more information, see Use Knative on ASM to deploy a serverless application.
Deploy Knative Serving
- To deploy Knative in an ACK cluster, see Deploy Knative.
- To deploy Knative in an ACK Serverless cluster, see Enable Knative.
Note
If you select Kourier as the service gateway for a one-click Knative deployment, uninstall the Kourier component after the installation is complete. To do this, go to the Clusters page in the ACK console, click the target cluster, and choose Applications > Knative. After you select Kourier as the gateway and the components are installed, go to the Components tab and uninstall the Kourier component in the Add-on Components section.
Enable Knative on ASM
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Ecosystem > Knative on ASM.
3. On the Knative on ASM page, click Enable Knative on ASM.
Object Storage Service (OSS) has been activated and a bucket has been created. For more information, see Activate OSS and Create buckets.

Step 1: Enable KServe on ASM

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Ecosystem > KServe on ASM.
On the KServe on ASM page, turn the Automatically install the CertManager component in the cluster switch on or off, and then click Enable KServe on ASM.

cert-manager is a certificate lifecycle management system that automates certificate issuance and deployment. KServe on ASM depends on the cert-manager component. When you install KServe, the cert-manager component is automatically installed.
- If cert-manager is not installed in your cluster, turn on the Automatically install the CertManager component in the cluster switch.
- If cert-manager is already installed in your data plane cluster, turn off the Automatically install the CertManager component in the cluster switch.

Step 2: Install Fluid and accelerate caching

Deploy the ack-fluid component, version 0.9.10 or later, in your cluster.
- If your data plane cluster is an ACK cluster, install the cloud-native AI suite and deploy the ack-fluid component.
  
  Note
  If you have open-source Fluid installed, uninstall it before you deploy the ack-fluid component.
  - If you have not installed the cloud-native AI suite, enable Fluid Data Acceleration during installation. For more information, see Deploy the cloud-native AI suite.
  - If you have already installed the cloud-native AI suite, log on to the ACK console. Go to the Applications > Cloud-native AI Suite page and deploy ack-fluid.
- If your data plane cluster is an ACK Serverless cluster, deploy the ack-fluid component. For more information, see the Deploy the control plane components of Fluid section of the Accelerate data access for Job applications topic.
Prepare an AI model and upload it to an OSS bucket.
1. Prepare the data from your trained AI model.
  
  This topic uses the BLOOM model, an open-source Transformer-based large language model (LLM) built on PyTorch, as an example. For more information about the model data, see Hugging Face.
2. Upload the downloaded model data files to an OSS bucket and note the storage path.
  
  The storage path format is oss://{bucket}/{path}. For example, if you create a bucket named fluid-demo and upload all model data files to the models/bloom directory within it, the storage path is oss://fluid-demo/models/bloom.
  
  Note
  You can use ossutil, a client tool provided by OSS, to upload data. For more information, see Install ossutil.
Create a namespace for the Fluid cache and AI service, and configure OSS access permissions.
1. Use kubectl to connect to the data plane cluster. For more information, see Connect to an ACK cluster by using kubectl.
2. Run the following command to create the kserve-fluid-demo namespace for the Fluid cache and KServe AI service:
```
kubectl create ns kserve-fluid-demo
```
3. Create a file named oss-secret.yaml with the following content.
  
  The fs.oss.accessKeyId and fs.oss.accessKeySecret fields are the AccessKey ID and secret for accessing OSS.
```
apiVersion: v1
kind: Secret
metadata:
  name: access-key
stringData:
  fs.oss.accessKeyId: xxx # Replace with an Alibaba Cloud AccessKey ID that can access OSS.
  fs.oss.accessKeySecret: xxx # Replace with the corresponding AccessKey secret.
```
4. Run the following command to apply the Secret and configure the OSS access credentials:
```
kubectl apply -f oss-secret.yaml -n kserve-fluid-demo
```

Declare the AI model data to be accessed in Fluid.

Submit a Dataset custom resource (CR) and a JindoRuntime CR. The Dataset CR defines the URL of the data in the external storage system, while the JindoRuntime CR describes the caching system and its configuration.

Create a file named oss-jindo.yaml with the following content.

In the Dataset CR, replace oss://{bucket}/{path} with the model data storage path you recorded in step 2.b, and replace {endpoint} with the OSS endpoint. For information about OSS endpoints in different regions, see Regions and endpoints.

oss-jindo.yaml

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: oss-data
spec:
  mounts:
  - mountPoint: "oss://{bucket}/{path}" # Replace with the storage path of the model data files.
    name: bloom-560m
    path: /bloom-560m
    options:
      fs.oss.endpoint: "{endpoint}"  # Replace with the actual OSS endpoint.
    encryptOptions:
      - name: fs.oss.accessKeyId
        valueFrom:
          secretKeyRef:
            name: access-key
            key: fs.oss.accessKeyId
      - name: fs.oss.accessKeySecret
        valueFrom:
          secretKeyRef:
            name: access-key
            key: fs.oss.accessKeySecret
  accessModes:
    - ReadOnlyMany
---
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
  name: oss-data
spec:
  replicas: 2
  tieredstore:
    levels:
      - mediumtype: SSD
        volumeType: emptyDir
        path: /mnt/ssd0/cache
        quota: 50Gi
        high: "0.95"
        low: "0.7"
  fuse:
    properties:
      fs.jindofsx.data.cache.enable: "true"
    args:
      - -okernel_cache
      - -oro
      - -oattr_timeout=7200
      - -oentry_timeout=7200
      - -ometrics_port=9089
    cleanPolicy: OnDemand

Run the following command to deploy the Dataset and JindoRuntime CRs:
```
kubectl create -f oss-jindo.yaml -n kserve-fluid-demo
```

Run the following command to check the deployment status of the Dataset and JindoRuntime:

kubectl get jindoruntime,dataset -n kserve-fluid-demo

Expected output:

NAME                                  MASTER PHASE   WORKER PHASE   FUSE PHASE   AGE
jindoruntime.data.fluid.io/oss-data   Ready          Ready          Ready        3m

NAME                             UFS TOTAL SIZE   CACHED   CACHE CAPACITY   CACHED PERCENTAGE   PHASE   AGE
dataset.data.fluid.io/oss-data   3.14GiB          0.00B    100.00GiB        0.0%                Bound   3m

The output shows that the PHASE of the Dataset is Bound and the FUSE PHASE of the JindoRuntime is Ready, which indicates that the deployment is successful.

Prefetch data with Fluid to improve access performance.

Create a file named oss-dataload.yaml with the following content:

apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
  name: oss-dataload
spec:
  dataset:
    name: oss-data
    namespace: kserve-fluid-demo
  target:
    - path: /bloom-560m
      replicas: 2

Run the following command to deploy the DataLoad CR and prefetch the data:
```
kubectl create -f oss-dataload.yaml -n kserve-fluid-demo
```
Run the following command to check the progress of the data prefetching:
```
kubectl get dataload -n kserve-fluid-demo
```
Expected output:
```
NAME           DATASET    PHASE      AGE     DURATION
oss-dataload   oss-data   Complete   1m      45s
```
The output shows that data prefetching takes about 45s. You must wait for the process to complete.

Step 3: Deploy the inference service

Create a file named oss-fluid-isvc.yaml based on your cluster type.

ACK cluster

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "fluid-bloom"
spec:
  predictor:
    timeout: 600
    minReplicas: 0
    containers:
      - name: kserve-container
        image: registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu
        resources:
          limits:
            cpu: "12"
            memory: 48Gi
            nvidia.com/gpu: 1 # If you use GPUs, specify the number required. Otherwise, you can omit this field.
          requests:
            cpu: "12"
            memory: 48Gi
        env:
          - name: STORAGE_URI
            value: "pvc://oss-data/bloom-560m"
          - name: MODEL_NAME
            value: "bloom"
            # Set this parameter to "True" if you use GPUs. Otherwise, set it to "False".
          - name: GPU_ENABLED
            value: "True"

ACK Serverless cluster

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "fluid-bloom"
  labels:
    alibabacloud.com/fluid-sidecar-target: "eci"
  annotations:
    k8s.aliyun.com/eci-use-specs : "ecs.gn6i-c16g1.4xlarge"  # Replace with the desired ECS instance type.
    knative.aliyun.com/reserve-instance-eci-use-specs: "ecs.gn6i-c16g1.4xlarge"   # Replace with the desired ECS instance type.
spec:
  predictor:
    timeout: 600
    minReplicas: 0
    containers:
      - name: kserve-container
        image: registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu
        resources:
          limits:
            cpu: "12"
            memory: 48Gi
          requests:
            cpu: "12"
            memory: 48Gi
        env:
          - name: STORAGE_URI
            value: "pvc://oss-data/bloom-560m"
          - name: MODEL_NAME
            value: "bloom"
            # Set this parameter to "True" if you use GPUs. Otherwise, set it to "False".
          - name: GPU_ENABLED
            value: "True"

Note

This example uses a large language model and requests significant resources (12 CPU cores, 48 GiB of memory). Adjust the resources field in the InferenceService configuration based on your cluster's workload.
This example sets the image field to the sample image registry.cn-hangzhou.aliyuncs.com/acs/kserve-fluid:bloom-gpu. This image provides interfaces for model loading and inference. View the code for this sample image in the KServe open-source community to customize your image. For more information, see the KServe Docker examples.

Run the following command to deploy the AI model inference service:
```
kubectl create -f oss-fluid-isvc.yaml -n kserve-fluid-demo
```

Run the following command to check the deployment status of the AI model inference service:

kubectl get inferenceservice -n kserve-fluid-demo

Expected output:

NAME          URL                                                READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION           AGE
fluid-bloom   http://fluid-bloom.kserve-fluid-demo.example.com   True           100                              fluid-bloom-predictor-00001   2d

When the READY field shows True, the AI model inference service has been deployed successfully.

Step 4: Access the inference service

Get the ASM ingress gateway address.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
3. In the Service Address section of the ingressgateway gateway, view and copy the ASM gateway service address.

Run the following command to access the sample AI model inference service, bloom. Replace {ASM gateway service address} with the ASM ingress gateway address you obtained in the previous step.

curl -v -H "Content-Type: application/json" -H "Host: fluid-bloom.kserve-fluid-demo.example.com" "http://{ASM gateway service address}:80/v1/models/bloom:predict" -d '{"prompt": "It was a dark and stormy night", "result_length": 50}'

Expected output:

*   Trying xxx.xx.xx.xx :80...
* Connected to xxx.xx.xx.xx  (xxx.xx.xx.xx ) port 80 (#0)
> POST /v1/models/bloom:predict HTTP/1.1
> Host: fluid-bloom-predictor.kserve-fluid-demo.example.com
> User-Agent: curl/7.84.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 65
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 227
< content-type: application/json
< date: Thu, 20 Apr 2023 09:49:00 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 1142
< 
{
  "result": "It was a dark and stormy night, and the wind was blowing in the\ndirection of the west. The wind was blowing in the direction of the\nwest, and the wind was blowing in the direction of the west. The\nwind was"
}
* Connection # 0 to host xxx.xx.xx.xx left intact

The output shows that the AI model inference service successfully generated text based on the sample input and returned an inference result.

Alibaba Cloud Service Mesh:Accelerate AI serving with KServe, Fluid, and ASM

Prerequisites

Step 1: Enable KServe on ASM

Step 2: Install Fluid and accelerate caching

Step 3: Deploy the inference service

ACK cluster

ACK Serverless cluster

Step 4: Access the inference service

Related topics