Deploy AI inference services in ACK clusters
Using the Arena CLI and AI workload scheduling in the ACK cloud-native AI suite, deploy trained models as inference services in Kubernetes clusters. ACK supports elastic scaling, GPU sharing, and performance monitoring to reduce operational costs. This topic describes how to deploy model inference services with ACK and the cloud-native AI suite.
NVIDIA Triton Server and TensorFlow Serving in ack-arena are free open source components provided by third-party open source communities or enterprises. You can choose to install the corresponding components and configure servers to deploy inference models as services, and then use the relevant model testing and optimization tools.
However, Alibaba Cloud is not responsible for the stability, service limits, and security compliance of third-party components. You shall pay close attention to the official websites of the third-party open source communities or enterprises and updates on code hosting platforms, and read and comply with the open source licenses. You are liable for any potential risks related to application development, maintenance, troubleshooting, and security due to the use of third-party components.
The cloud-native AI suite supports these inference service types.
|
Inference service type |
Description |
References |
|
Shared GPU inference tasks |
Use Arena to run multiple inference tasks on a shared GPU to improve GPU utilization. |
|
|
TensorFlow model inference services |
Use Arena and TensorFlow Serving to deploy TensorFlow models as inference services. |
|
|
PyTorch model inference services |
Use NVIDIA Triton Inference Server or TorchServe to deploy PyTorch models as inference services. |
|
|
Containerized elastic inference services |
Deploy elastic inference services on Elastic Compute Service (ECS) or Elastic Container Instance to improve elasticity and reduce costs. |