Alibaba Cloud Service Mesh:Deploy AI model inference services with KServe and ASM

Step 1: Install the KServe component

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the target ASM instance. In the navigation pane on the left, choose Ecosystem > KServe on ASM.

3. On the KServe on ASM page, click Enable KServe on ASM.

   KServe depends on cert-manager. When you install KServe, the cert-manager component is installed automatically. If you want to use a self-managed cert-manager, turn off Automatically install the CertManager component in the cluster.

Step 2: Get the ASM gateway address

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.

3. On the Ingress Gateway page, find and record the Service address of the target gateway.

Step 3: Create an inference service

This topic uses a pre-trained scikit-learn model for testing.

1. Connect to the data plane cluster using kubectl and run the following command to create a namespace for KServe resources:

   kubectl create namespace kserve-test

2. Create an InferenceService.

   1. Create a file named isvc.yaml with the following content:

      apiVersion: "serving.kserve.io/v1beta1"
      kind: "InferenceService"
      metadata:
        name: "sklearn-iris"
      spec:
        predictor:
          model:
            modelFormat:
              name: sklearn
            storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"

   2. Run the following command to create the InferenceService in the kserve-test namespace:

      kubectl apply -f isvc.yaml -n kserve-test

3. Run the following command to check the status of the sklearn-iris InferenceService:

   kubectl get inferenceservices sklearn-iris -n kserve-test

   Expected output:

   NAME           URL                                           READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION            AGE
   sklearn-iris   http://sklearn-iris.kserve-test.example.com   True           100                              sklearn-iris-predictor-00001   3h26m

   READY is True, indicating that the InferenceService sklearn-iris is successfully installed.

4. Optional: View the virtual service and Gateway.

   Creating the sklearn-iris InferenceService automatically creates a corresponding virtual service and Gateway.

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Traffic Management Center > VirtualService.

   3. On the VirtualService page, click the icon next to Namespaces, select Namespaces from the Namespace drop-down list, and view the new virtual service.

   4. In the navigation pane on the left, choose ASM Gateways > Gateway.

   5. On the Gateway page, select Namespaces from the Namespace drop-down list to view the new Gateway.

Step 4: Access the model service

The following examples are for Linux and macOS environments.

1. Run the following command to create a model input file:

   cat <<EOF > "./iris-input.json"
   {
     "instances": [
       [6.8,  2.8,  4.8,  1.4],
       [6.0,  3.4,  4.5,  1.6]
     ]
   }
   EOF

2. Test access via the ASM gateway.

   1. Run the following command to obtain the SERVICE_HOSTNAME:

      SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -n kserve-test -o jsonpath='{.status.url}' | cut -d "/" -f 3)
      echo $SERVICE_HOSTNAME

      Expected output:

      sklearn-iris.kserve-test.example.com

   2. Run the following command to access the model service using the ASM gateway address that you obtained in Step 2:

      ASM_GATEWAY="XXXX" # Replace XXXX with your ASM gateway address.
      curl  -H "Host: ${SERVICE_HOSTNAME}" http://${ASM_GATEWAY}:80/v1/models/sklearn-iris:predict -d @./iris-input.json

      Expected output:

      {"predictions": [1, 1]}

3. Load test the model service.

   1. Run the following command to deploy a client application for load testing:

      kubectl create -f https://alibabacloudservicemesh.oss-cn-beijing.aliyuncs.com/kserve/v0.7/loadtest.yaml

   2. Run the following command to view the pod name:

      kubectl get pod

      Expected output:

      NAME                                                       READY   STATUS      RESTARTS   AGE
      load-testxhwtq-pj9fq                                       0/1     Completed   0          3m24s
      sklearn-iris-predictor-00001-deployment-857f9bb56c-vg8tf   2/2     Running     0          51m

   3. Run the following command to view the load test logs:

      kubectl logs load-testxhwtq-pj9fq # Replace the pod name with the actual pod name of the load test application.

      Expected output:

      Requests      [total, rate, throughput]         30000, 500.02, 500.01
      Duration      [total, attack, wait]             59.999s, 59.998s, 1.352ms
      Latencies     [min, mean, 50, 90, 95, 99, max]  1.196ms, 1.463ms, 1.378ms, 1.588ms, 1.746ms, 2.99ms, 18.873ms
      Bytes In      [total, mean]                     690000, 23.00
      Bytes Out     [total, mean]                     2460000, 82.00
      Success       [ratio]                           100.00%
      Status Codes  [code:count]                      200:30000
      Error Set:

Related documentation

• To accelerate model loading for data-intensive, cloud-native applications such as big data and AI, see Accelerate data access for AI Serving by using KServe and Fluid on ASM.

• To convert raw input data into the format required by the model server, see Deploy a transformer by using an InferenceService.

• For large-scale, high-density, and frequently changing model services, you can use Model Service Mesh to manage, deploy, and schedule multiple model services. For more information, see Model Service Mesh.