KServe provides default Prometheus metrics to monitor model service performance and health. This topic explains how to configure Prometheus monitoring for KServe.
Prerequisites
-
Arena client version 0.9.15 or later is installed.
-
The ack-kserve component is installed.
-
Alibaba Cloud Prometheus monitoring is enabled.
Step 1: Deploy KServe application
-
Deploy a Scikit-learn KServe application:
arena serve kserve \ --name=sklearn-iris \ --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/ai-sample/kserve-sklearn-server:v0.12.0 \ --cpu=1 \ --memory=200Mi \ --enable-prometheus=true \ --metrics-port=8080 \ "python -m sklearnserver --model_name=sklearn-iris --model_dir=/models --http_port=8080"Expected output:
service/sklearn-iris-metric-svc created # A service named sklearn-iris-metric-svc is created. inferenceservice.serving.kserve.io/sklearn-iris created # The KServe InferenceService resource sklearn-iris is created. servicemonitor.monitoring.coreos.com/sklearn-iris-svcmonitor created # A ServiceMonitor resource is created to integrate with the Prometheus monitoring system and collect monitoring data exposed by the sklearn-iris-metric-svc service. INFO[0004] The Job sklearn-iris has been submitted successfully # The job is successfully submitted to the cluster. INFO[0004] You can run `arena serve get sklearn-iris --type kserve -n default` to check the job statusArena deployed a KServe service with a scikit-learn model and Prometheus monitoring.
-
Write the inference input to an
./iris-input.jsonfile:cat <<EOF > "./iris-input.json" { "instances": [ [6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6] ] } EOF -
Retrieve the NGINX Ingress gateway IP address and InferenceService hostname:
NGINX_INGRESS_IP=`kubectl -n kube-system get svc nginx-ingress-lb -ojsonpath='{.status.loadBalancer.ingress[0].ip}'` SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -o jsonpath='{.status.url}' | cut -d "/" -f 3) -
Use the Hey stress testing tool to send repeated requests and generate monitoring data:
NoteSee Hey.
hey -z 2m -c 20 -m POST -host $SERVICE_HOSTNAME -H "Content-Type: application/json" -D ./iris-input.json http://${NGINX_INGRESS_IP}:80/v1/models/sklearn-iris:predictExpected output:
The output shows performance metrics such as processing speed, throughput, and response latency.
-
(Optional) Verify that application metrics are exposed correctly.
Collect monitoring metrics from the
sklearn-irisPod locally, without logging on to the Pod or exposing its port.-
Forward port 8080 from the sklearn-iris Pod (specified by
$POD_NAME) to your local host:# Get the Pod name. POD_NAME=`kubectl get po|grep sklearn-iris |awk -F ' ' '{print $1}'` # Forward port 8080 of the pod to the local host. kubectl port-forward pod/$POD_NAME 8080:8080Expected output:
Forwarding from 127.0.0.1:8080 -> 8080 Forwarding from [::1]:8080 -> 8080Both IPv4 and IPv6 connections are forwarded to the Pod's port 8080.
-
Open the following URL in a browser to view Pod metrics:
http://localhost:8080/metricsExpected output:
The output confirms that metrics are exposed and requests are forwarded to the Pod application.
-
Step 2: Query KServe metrics
-
Log on to the ARMS console.
-
In the left-side navigation pane, click Integration Management, and then click Query Dashboards.
-
On the Dashboards page, click the Kubernetes Pod dashboard to open the Grafana page.
-
In the left-side navigation pane, click Explore. Enter the
request_predict_seconds_bucketquery to view metric values.NoteData may take up to 5 minutes to appear.
The graph displays a line chart of the
request_predict_seconds_bucketmetric over time.
FAQ
Question
How can I verify that data for the request_predict_seconds_bucket metric has been collected successfully?
Solution
-
Log on to the ARMS console.
-
In Connected Environments, on the Container Service tab, click the target container environment, and then click the Self-Monitoring tab.
-
In the left-side navigation pane, click Targets. If
default/sklearn-iris-svcmonitor/0 (1/1 up)is displayed, metric collection is successful.
References
For default KServe metrics, see KServe Prometheus Metrics.