All Products
Search

This Product

    Container Service for Kubernetes:Deploy an inference service with KServe

    Document Center

    Container Service for Kubernetes:Deploy an inference service with KServe

    Last Updated:Jun 20, 2026

    You can use KServe on ACK Knative or to deploy AI models as serverless inference services. This allows you to implement features such as auto-scaling, versioning, and canary releases.

    Step 1: Install and configure the KServe add-on

    To integrate KServe seamlessly with Knative's ALB Ingress or Kourier gateway , you must first install the KServe add-on. Then, modify its default settings to disable the built-in Istio VirtualService.

    1. Install the KServe add-on.

      1. Log on to the ACK console. In the left navigation pane, click Clusters.

      2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Knative.

      3. On the Add-ons tab, find and deploy the KServe add-on in the Add-on Components section.

    2. Disable Istio VirtualService creation.

      Edit the inferenceservice-config ConfigMap and set disableIstioVirtualHost to true.

      kubectl get configmap inferenceservice-config -n kserve -o yaml \
| sed 's/"disableIstioVirtualHost": false/"disableIstioVirtualHost": true/g' \
| kubectl apply -f -

      Expected output:

      configmap/inferenceservice-config configured

    3. Verify the configuration.

      kubectl get configmap inferenceservice-config -n kserve -o yaml \
| grep '"disableIstioVirtualHost":' \
| tail -n1 \
| awk -F':' '{gsub(/[ ,]/,"",$2); print $2}'

      An output of true indicates the configuration is updated.

    4. Restart the KServe controller to apply the configuration change.

      kubectl rollout restart deployment kserve-controller-manager -n kserve

    Step 2: Deploy the InferenceService

    This example deploys a scikit-learn classification model trained on the Iris dataset. The service receives four feature values that represent the dimensions of a flower and predicts its species.

    The input consists of four sequential numerical features:

    1. Sepal length

    2. Sepal width

    3. Petal length

    4. Petal width

    The output is an index representing the predicted class:

    • 0: Iris setosa

    • 1: Iris versicolour

    • 2: Iris virginica

    1. Create an inferenceservice.yaml file to deploy the inference service.

      apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      # The format of the model. In this case, sklearn.
      modelFormat:
        name: sklearn
      image: "kube-ai-registry.cn-shanghai.cr.aliyuncs.com/ai-sample/kserve-sklearn-server:v0.12.0"
      command:
      - sh
      - -c
      - "python -m sklearnserver --model_name=sklearn-iris --model_dir=/models --http_port=8080"

    2. Deploy the InferenceService.

      kubectl apply -f inferenceservice.yaml

    3. Check the service status.

      kubectl get inferenceservices sklearn-iris

      In the output, True in the READY column indicates that the service is ready.

      NAME           URL                                                         READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                    AGE
sklearn-iris   http://sklearn-iris-predictor-default.default.example.com   True           100                              sklearn-iris-predictor-default-00001   51s

    Step 3: Access the service

    Send an inference request to the service through the cluster's ingress gateway.

    1. On the Knative page, go to the Services tab. Obtain the Gateway of the ALB or Kourier gateway and the Default Domain of the service.

      This example uses an ALB Ingress. The user interface for a Kourier gateway is similar.

      At the top of the page, the gateway address is displayed, such as alb-xxx.aliyuncsslb.com. In the service list, the status of sklearn-iris-predictor is Successful and its default domain name is sklearn-iris-predictor.default.example.com.

    2. Prepare the request data.

      In your local terminal, create a ./iris-input.json file that contains two arrays. Each array represents a sample to be predicted.

      cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

    3. In your local terminal, send an inference request to access the service.

      Replace ${INGRESS_DOMAIN} with the Gateway that you obtained in Step 1.

      curl -H "Content-Type: application/json" -H "Host: sklearn-iris-predictor.default.example.com" "http://${INGRESS_DOMAIN}/v1/models/sklearn-iris:predict" -d @./iris-input.json

      The output indicates that the model predicts class index 1 (Iris versicolour) for both input samples.

      {"predictions":[1,1]}

    Billing

    The KServe and Knative components are free of charge. However, you are charged for the computing resources (such as ECS and Elastic Container Instance) and network resources (such as ALB and CLB) that you use. For more information, see Cloud product resource fees.

    FAQ

    InferenceService in Not Ready state

    To troubleshoot the issue:

    1. Run kubectl describe inferenceservice <yourServiceName> and check the events for error messages.

    2. Run kubectl get pods to check whether any Pods associated with the service are in the Error or CrashLoopBackOff state. The names of these Pods typically start with the service name.

    3. If a Pod is in an abnormal state, view the logs of the model service container by running kubectl logs <pod-name> -c kserve-container. In the logs, check for model loading failures caused by issues such as network errors or an incorrect model file format.

    Deploying a custom model

    You can upload the model file to an OSS bucket. When you create the InferenceService, replace the value of the spec.predictor.model.storageUri field with the URI of the model file. You must also set the modelFormat parameter to a valid value based on the model framework, such as tensorflow, pytorch, or onnx.

    Configuring GPU resources

    If your model requires a GPU for inference, you can add the resources field to the predictor section of the InferenceService YAML file to request GPU resources. For example:

    For more information about how to use GPU resources in Knative, see Use GPU resources.
    spec:
  predictor:
    resources:
      requests:
        nvidia.com/gpu: "1"
      limits:
        nvidia.com/gpu: "1"

    Related documentation