KServe is an open-source, cloud-native model serving platform that simplifies deploying and running machine learning models on Kubernetes. It supports multiple frameworks, provides elastic scaling, and uses declarative APIs defined in simple YAML files to make model service configuration and management easier.
Framework
KServe manages model services through CRDs with standardized data plane protocols for frameworks such as TensorFlow, XGBoost, scikit-learn, PyTorch, and Hugging Face Transformer/LLM. It abstracts AutoScaling, networking, health checking, and server configuration to enable GPU auto scaling, Scale to Zero, and Canary Rollouts.
See KServe on GitHub.
Deployment modes
KServe supports three deployment modes: Raw Deployment, Serverless, and ModelMesh. Feature support varies by mode.
|
Deployment mode |
Description |
References |
|
Raw Deployment |
The simplest mode, requiring only cert-manager and gateways. Supports features such as AutoScaling, Prometheus monitoring, Canary Rollouts with specific gateways, and GPU auto scaling. |
|
|
Serverless |
Requires cert-manager, gateways, and Knative. Supports features such as autoscaling, Scale to Zero, Canary Rollouts, and GPU autoscaling. |
|
|
ModelMesh |
Requires cert-manager, Knative, and ModelMesh. For example, it can run with Alibaba Cloud Service Mesh (ASM).Service Mesh (ASM), for example. Supports features such as AutoScaling, Scale to Zero, Canary Rollouts, and GPU auto scaling. |
N/A |
ack-kserve installation
To deploy and manage ack-kserve in an ACK cluster, see Install the ack-kserve add-on.