<comment commentTypeId=

ack-kserve is a Container Service for Kubernetes (ACK) add-on built on open source KServe with Alibaba Cloud optimizations. It simplifies the deployment and management of machine learning models in ACK Edge clusters.

KServe is an open source project that provides a declarative API based on YAML files to simplify the deployment and management of machine learning models on Kubernetes. The ack-kserve add-on integrates with Alibaba Cloud services such as storage, logging, and networking to simplify KServe deployment and operations.

Prerequisites

An ACK Edge cluster of version 1.22 or later is created. For instructions, see Create an ACK Edge cluster.
The NGINX Ingress controller is installed in the ACK Edge cluster. For instructions, see Deploy an Ingress controller in an ACK Edge cluster.
The kubectl client is installed on your local machine and configured to connect to the ACK Edge cluster.

Step 1: Install cert-manager

ack-kserve depends on cert-manager for Transport Layer Security (TLS) certificate management.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, click Deploy. In Basic Information, enter cert-manager for Application Name, select cert-manager in the Chart section, and click Next.
In Parameters, review Chart Version and Parameters, and then click OK.

After deployment, cert-manager appears on the Helm page.

Verify that all cert-manager pods are ready:

kubectl -n cert-manager get po

All pods must show 1/1 in the READY column and Running in the STATUS column:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7f4bb44d5b-jrrfn              1/1     Running   0          23h
cert-manager-cainjector-79544456cc-qp5pp   1/1     Running   0          23h
cert-manager-webhook-f74ccb647-7m5dt       1/1     Running   0          23h

Step 2: Install the ack-kserve add-on

By default, ack-kserve is deployed in RawDeployment mode and integrates with the NGINX Ingress controller. The following steps install the add-on with this default configuration.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, click Deploy. In Basic Information, enter kserve for Application Name, select ack-kserve in the Chart section, and click Next.
In Parameters, review Chart Version and Parameters, and then click OK.

After deployment, the kserve release appears on the Helm page.
Configure the ingress class for KServe.

On the Helm page, click kserve, and then click inferenceservice-config. Click Edit YAML and set the ingressClassName field in the YAML file to the same value as the ingressClassResource.name that you specified when you deployed the NGINX Ingress controller. For instructions, see Deploy an Ingress controller in an ACK Edge cluster.
Verify that the ack-kserve add-on is running:
```
kubectl get pod -n kserve
```
If all pods show running in the STATUS column, the ack-kserve add-on is installed.

(Optional) Step 3: View or update ack-kserve

View details of ack-kserve

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, find kserve and click View Details in the Actions column.

The details page shows the Basic Information, Parameters, and History tabs.

Update ack-kserve

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, find kserve and click Update in the Actions column.
In the Update Release panel, change the chart version or modify parameters, and then click OK.

After the update completes, the kserve release runs with the updated chart version or parameters.

(Optional) Step 4: Uninstall the ack-kserve add-on

Uninstall the add-ons in the following order to avoid orphaned resources:

Delete KServe custom resources.
Delete the KServe CustomResourceDefinitions (CRDs).
Uninstall the ack-kserve add-on.
Uninstall cert-manager.
Delete the cert-manager CRDs.

Delete KServe custom resources and CRDs

Deleting a CRD automatically deletes all associated custom resources, and custom resources cannot be restored after deletion. Verify that your InferenceService resources are no longer needed before you proceed.

Back up and delete all InferenceService custom resources:

# List all InferenceService resources in the cluster
kubectl get isvc --all-namespaces

# Back up all resources to a file
kubectl get isvc --all-namespaces -oyaml > isvc.yaml.bak

# Delete all resources
kubectl delete isvc --all

Delete the KServe CRDs:

Important

Delete all custom resources before you delete a CRD. If you delete a CRD first, the associated custom resources are deleted without a backup.

kubectl delete crd clusterservingruntimes.serving.kserve.io
kubectl delete crd clusterstoragecontainers.serving.kserve.io
kubectl delete crd inferencegraphs.serving.kserve.io
kubectl delete crd inferenceservices.serving.kserve.io
kubectl delete crd predictors.serving.kserve.io
kubectl delete crd servingruntimes.serving.kserve.io
kubectl delete crd trainedmodels.serving.kserve.io

Uninstall ack-kserve

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, find kserve, the Helm release of the ack-kserve add-on, and click Delete in the Actions column. In the dialog box, click OK.

Uninstall cert-manager

Ensure that no other cluster add-ons depend on cert-manager before you uninstall it. Removing cert-manager while it is still in use interrupts the workloads that depend on it.

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left-side navigation pane, choose Applications > Helm.
On the Helm page, find cert-manager and click Delete in the Actions column. In the dialog box, click OK.

Delete the cert-manager CRDs:

kubectl delete crd certificaterequests.cert-manager.io
kubectl delete crd certificates.cert-manager.io
kubectl delete crd challenges.acme.cert-manager.io
kubectl delete crd clusterissuers.cert-manager.io
kubectl delete crd issuers.cert-manager.io
kubectl delete crd orders.acme.cert-manager.io

Next steps

After you install the ack-kserve add-on, you can create InferenceService custom resources to deploy your machine learning models as inference services in the ACK Edge cluster.

Troubleshooting

TLS certificate verification failed during ack-kserve installation

The ack-kserve installation fails with the following error:

failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority

ack-kserve requires cert-manager to be fully ready. This error occurs when cert-manager is not installed or its pods are not yet running.

Check whether cert-manager is installed:
```
kubectl get crd | grep certificates.cert-manager.io
```
Expected output if cert-manager is installed:
```
certificates.cert-manager.io                         2024-05-06T07:09:17Z
```
If there is no output, install cert-manager first.

Check whether the cert-manager pods are ready:

kubectl -n cert-manager get po

If cert-manager is ready, all pods show 1/1 in the READY column:

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-7f4bb44d5b-jrrfn              1/1     Running   0          23h
cert-manager-cainjector-79544456cc-qp5pp   1/1     Running   0          23h
cert-manager-webhook-f74ccb647-7m5dt       1/1     Running   0          23h

If all pods are ready, uninstall ack-kserve and reinstall it.