All Products
Search

This Product

    Container Service for Kubernetes:Deploy an inference service with KServe

    Document Center

    Container Service for Kubernetes:Deploy an inference service with KServe

    Last Updated:Jun 24, 2026

    Use KServe on ACK Knative to deploy AI models as serverless inference services. This approach provides key features such as auto-scaling, version management, and canary releases.

    Step 1: Install and configure the KServe component

    To ensure seamless integration with Knative's ALB Ingress or Kourier gateway, install the KServe component and modify its settings to disable the creation of built-in Istio VirtualService resources.

    1. Install the KServe component.

      1. Log on to the ACK console. In the left navigation pane, click Clusters.

      2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Knative.

      3. On the Add-ons tab, find and deploy the KServe component in the Add-on Components section.

    2. Disable Istio VirtualHost creation.

      Edit theinferenceservice-config ConfigMap to set disableIstioVirtualHost to true.

      kubectl get configmap inferenceservice-config -n kserve -o yaml \
| sed 's/"disableIstioVirtualHost": false/"disableIstioVirtualHost": true/g' \
| kubectl apply -f -

      Expected output:

      configmap/inferenceservice-config configured

    3. Verify the configuration.

      kubectl get configmap inferenceservice-config -n kserve -o yaml \
| grep '"disableIstioVirtualHost":' \
| tail -n1 \
| awk -F':' '{gsub(/[ ,]/,"",$2); print $2}'

      An output oftrue indicates that the configuration is updated.

    4. Restart the KServe controller to apply the configuration change.

      kubectl rollout restart deployment kserve-controller-manager -n kserve

    Step 2: Deploy the InferenceService

    This example deploys a scikit-learn classification model trained on the Iris flower dataset. The model receives four flower measurements as input and predicts the corresponding species.

    The input is an array of four ordered numerical features:

    1. Sepal Length

    2. Sepal Width

    3. Petal Length

    4. Petal Width

    The output is the index of the predicted class:

    • 0: Iris setosa

    • 1: Iris versicolour

    • 2: Iris virginica

    1. Create an inferenceservice.yaml file to deploy the inference service.

      apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-iris"
spec:
  predictor:
    model:
      # The format of the model. In this case, sklearn.
      modelFormat:
        name: sklearn
      image: "kube-ai-registry.cn-shanghai.cr.aliyuncs.com/ai-sample/kserve-sklearn-server:v0.12.0"
      command:
      - sh
      - -c
      - "python -m sklearnserver --model_name=sklearn-iris --model_dir=/models --http_port=8080"

    2. Deploy the InferenceService.

      kubectl apply -f inferenceservice.yaml

    3. Check the service status.

      kubectl get inferenceservices sklearn-iris

      A READY status ofTrue indicates that the service is running properly.

      NAME           URL                                                         READY   PREV   LATEST   PREVROLLEDOUTREVISION   LATESTREADYREVISION                    AGE
sklearn-iris   http://sklearn-iris-predictor-default.default.example.com   True           100                              sklearn-iris-predictor-default-00001   51s

    Step 3: Access the service

    Send an inference request to the service through the cluster's ingress gateway.

    1. On the Knative page, on the Services tab, obtain the Gateway of the ALB Ingress or Kourier gateway and the Default Domain of the service.

      This example uses an ALB Ingress. The UI for the Kourier gateway is similar.

      In the service list, the status of sklearn-iris-predictor is Ready, and the default domain is sklearn-iris-predictor.default.example.com. The gateway address of the ALB Ingress is displayed at the top of the page. Record the address and default domain.

    2. Prepare the request data.

      In your local terminal, create a file named./iris-input.json that contains two arrays. Each array represents a sample for prediction.

      cat <<EOF > "./iris-input.json"
{
  "instances": [
    [6.8,  2.8,  4.8,  1.4],
    [6.0,  3.4,  4.5,  1.6]
  ]
}
EOF

    3. Send an inference request from your local terminal to access the service.

      Replace${INGRESS_DOMAIN} with the Gateway from step 1 of this section.

      curl -H "Content-Type: application/json" -H "Host: sklearn-iris-predictor.default.example.com" "http://${INGRESS_DOMAIN}/v1/models/sklearn-iris:predict" -d @./iris-input.json

      The output indicates that the model predicts class index 1 for both input samples, which corresponds to Iris versicolour.

      {"predictions":[1,1]}

    Billing

    The KServe and Knative components are free of charge. However, you are charged for the computing resources, such as ECS and ECI instances, and network resources, such as ALB and CLB instances, that you use. For more information, see Pricing of cloud resources.

    FAQ

    Why is my InferenceService stuck in the Not Ready state?

    To troubleshoot this issue, perform the following steps:

    1. Run the kubectl describe inferenceservice <yourServiceName> command to check the events for any error messages.

    2. Run kubectl get pods to check if any Pods related to the service (typically starting with the service name) are in an Error or CrashLoopBackOff state.

    3. If a Pod is in an abnormal state, run the kubectl logs <pod-name> -c kserve-container command to check the logs of the model serving container for model loading failures, such as being unable to download the model due to network issues or using an incorrect model file format.

    How can I deploy my own trained model?

    You can upload the model file to an OSS Bucket. When you create an InferenceService, set the spec.predictor.model.storageUri field to the URI of the model file. At the same time, correctly set the modelFormat field based on the model framework (for example, tensorflow, pytorch, or onnx).

    How do I configure GPU resources for the model service?

    If a model requires a GPU for inference, you can add the resources field to the predictor in the InferenceService YAML file to request GPU resources. The following is an example.

    For more information about how to use GPU resources in Knative, see Use GPU resources.
    spec:
  predictor:
    resources:
      requests:
        nvidia.com/gpu: "1"
      limits:
        nvidia.com/gpu: "1"

    Related topics