Container Service for Kubernetes:Konfigurasikan kebijakan auto scaling untuk layanan dengan KServe

Pilih jenis penskalaan

Prasyarat

• Arena client versi 0.9.15 atau yang lebih baru telah diinstal.

• Komponen ack-kserve telah diinstal.

Konfigurasi skalabilitas elastis berbasis CPU/memori

HPA menyesuaikan jumlah replika Pod dalam mode Raw Deployment berdasarkan pemanfaatan CPU atau memori. Contoh ini menggunakan model sklearn-iris dengan penskalaan berbasis CPU.

Lihat Horizontal Pod Autoscaling dalam dokumentasi Kubernetes.

1. Kirim layanan inferensi dengan parameter penskalaan.

   Parameter	Description
   --scale-metric	Metrik penskalaan. Nilai yang valid: cpu, memory.
   --scale-target	Ambang batas penskalaan, dalam persentase.
   --min-replicas	Jumlah minimum replika. Harus lebih besar dari 0.
   --max-replicas	Jumlah maksimum replika. Harus lebih besar dari --min-replicas.

   arena serve kserve \
       --name=sklearn-iris \
       --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/ai-sample/kserve-sklearn-server:v0.12.0 \
       --cpu=1 \
       --memory=200Mi \
       --scale-metric=cpu \
       --scale-target=10 \
       --min-replicas=1 \
       --max-replicas=10 \
       "python -m sklearnserver --model_name=sklearn-iris --model_dir=/models --http_port=8080"

   Output yang diharapkan:

   inferenceservice.serving.kserve.io/sklearn-iris created
   INFO[0002] The Job sklearn-iris has been submitted successfully
   INFO[0002] You can run `arena serve get sklearn-iris --type kserve -n default` to check the job status

2. Buat file bernama iris-input.json dengan konten berikut:

   cat <<EOF > "./iris-input.json"
   {
     "instances": [
       [6.8,  2.8,  4.8,  1.4],
       [6.0,  3.4,  4.5,  1.6]
     ]
   }
   EOF

3. Uji layanan inferensi.

   # Dapatkan IP load balancer dari layanan nginx-ingress-lb di namespace kube-system.
   NGINX_INGRESS_IP=`kubectl -n kube-system get svc nginx-ingress-lb -ojsonpath='{.status.loadBalancer.ingress[0].ip}'`
   # Dapatkan hostname dari InferenceService sklearn-iris.
   SERVICE_HOSTNAME=$(kubectl get inferenceservice sklearn-iris -o jsonpath='{.status.url}' | cut -d "/" -f 3)
   # Kirim permintaan prediksi menggunakan file iris-input.json.
   curl -H "Host: $SERVICE_HOSTNAME" -H "Content-Type: application/json" \
        http://$NGINX_INGRESS_IP:80/v1/models/sklearn-iris:predict -d @./iris-input.json

   Output yang diharapkan:

   {"predictions":[1,1]}%

4. Jalankan uji stres untuk memicu penskalaan.

   Catatan

   Langkah ini menggunakan alat uji stres Hey.

   hey -z 2m -c 20 -m POST -host $SERVICE_HOSTNAME -H "Content-Type: application/json" -D ./iris-input.json http://${NGINX_INGRESS_IP}:80/v1/models/sklearn-iris:predict

5. Saat uji stres berjalan, periksa status penskalaan HPA di terminal terpisah.

   kubectl describe hpa sklearn-iris-predictor

   Perluas untuk melihat status penskalaan layanan

   Name:                                                  sklearn-iris-predictor
   Namespace:                                             default
   Labels:                                                app=isvc.sklearn-iris-predictor
                                                          arena.kubeflow.org/uid=3399d840e8b371ed7ca45dda29debeb1
                                                          chart=kserve-0.1.0
                                                          component=predictor
                                                          heritage=Helm
                                                          release=sklearn-iris
                                                          serving.kserve.io/inferenceservice=sklearn-iris
                                                          servingName=sklearn-iris
                                                          servingType=kserve
   Annotations:                                           arena.kubeflow.org/username: kubecfg:certauth:admin
                                                          serving.kserve.io/deploymentMode: RawDeployment
   CreationTimestamp:                                     Sat, 11 May 2024 17:15:47 +0800
   Reference:                                             Deployment/sklearn-iris-predictor
   Metrics:                                               ( current / target )
     resource cpu on pods  (as a percentage of request):  0% (2m) / 10%
   Min replicas:                                          1
   Max replicas:                                          10
   Behavior:
     Scale Up:
       Stabilization Window: 0 seconds
       Select Policy: Max
       Policies:
         - Type: Pods     Value: 4    Period: 15 seconds
         - Type: Percent  Value: 100  Period: 15 seconds
     Scale Down:
       Select Policy: Max
       Policies:
         - Type: Percent  Value: 100  Period: 15 seconds
   Deployment pods:       10 current / 10 desired
   Conditions:
     Type            Status  Reason               Message
     ----            ------  ------               -------
     AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
     ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
     ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
   Events:
     Type    Reason             Age                  From                       Message
     ----    ------             ----                 ----                       -------
     Normal  SuccessfulRescale  38m                  horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
     Normal  SuccessfulRescale  28m                  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
     Normal  SuccessfulRescale  27m                  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

   Bagian Events menunjukkan bahwa HPA melakukan scale-out menjadi 8 replika saat beban tinggi, lalu melakukan scale-in menjadi 7 dan 1 seiring penurunan beban.

Konfigurasi skalabilitas elastis berbasis pemanfaatan GPU

Jenis ini menggunakan HPA dalam mode Raw Deployment dengan komponen ack-alibaba-cloud-metrics-adapter untuk mengekspos metrik GPU kustom. Lihat Horizontal pod autoscaling berdasarkan metrik Alibaba Cloud Prometheus.

Contoh ini mengonfigurasi auto scaling berbasis pemanfaatan GPU.

1. Siapkan data model Qwen-7B-Chat-Int8. Lihat Deploy a vLLM inference service.

2. Konfigurasi metrik GPU kustom. Lihat Implement elastic scaling based on GPU metrics.

3. Terapkan layanan inferensi vLLM dengan penskalaan berbasis GPU.

   arena serve kserve \
       --name=qwen \
       --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:0.4.1 \
       --gpus=1 \
       --cpu=4 \
       --memory=12Gi \
       --scale-metric=DCGM_CUSTOM_PROCESS_SM_UTIL \
       --scale-target=50 \
       --min-replicas=1 \
       --max-replicas=2 \
       --data="llm-model:/mnt/models/Qwen-7B-Chat-Int8" \
       "python3 -m vllm.entrypoints.openai.api_server --port 8080 --trust-remote-code --served-model-name qwen --model /mnt/models/Qwen-7B-Chat-Int8 --gpu-memory-utilization 0.95 --quantization gptq --max-model-len=6144"

   Output yang diharapkan:

   inferenceservice.serving.kserve.io/qwen created
   INFO[0002] The Job qwen has been submitted successfully
   INFO[0002] You can run `arena serve get qwen --type kserve -n default` to check the job status

4. Uji layanan inferensi.

   # Dapatkan alamat IP Nginx Ingress.
   NGINX_INGRESS_IP=$(kubectl -n kube-system get svc nginx-ingress-lb -ojsonpath='{.status.loadBalancer.ingress[0].ip}')
   # Dapatkan hostname dari InferenceService.
   SERVICE_HOSTNAME=$(kubectl get inferenceservice qwen -o jsonpath='{.status.url}' | cut -d "/" -f 3)
   # Kirim permintaan uji ke endpoint chat completions vLLM.
   curl -H "Host: $SERVICE_HOSTNAME" -H "Content-Type: application/json" http://$NGINX_INGRESS_IP:80/v1/chat/completions -d '{"model": "qwen", "messages": [{"role": "user", "content": "Test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

   Output yang diharapkan:

   {"id":"cmpl-77088b96abe744c89284efde2e779174","object":"chat.completion","created":1715590010,"model":"qwen","choices":[{"index":0,"message":{"role":"assistant","content":"OK, what do you need to test?<|im_end|>"},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":20,"completion_tokens":10}}%

5. Jalankan uji stres untuk memicu penskalaan berbasis GPU.

   Catatan

   Langkah ini menggunakan alat uji stres Hey.

   hey -z 2m -c 5 -m POST -host $SERVICE_HOSTNAME -H "Content-Type: application/json" -d '{"model": "qwen", "messages": [{"role": "user", "content": "Test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}' http://$NGINX_INGRESS_IP:80/v1/chat/completions

6. Saat uji stres berjalan, periksa status penskalaan HPA di terminal terpisah.

   kubectl describe hpa qwen-hpa

   Perluas untuk melihat status penskalaan qwen-hpa

   Name:                                     qwen-hpa
   Namespace:                                default
   Labels:                                   <none>
   Annotations:                              <none>
   CreationTimestamp:                        Tue, 14 May 2024 14:57:03 +0800
   Reference:                                Deployment/qwen-predictor
   Metrics:                                  ( current / target )
     "DCGM_CUSTOM_PROCESS_SM_UTIL" on pods:  0 / 50
   Min replicas:                             1
   Max replicas:                             2
   Deployment pods:                          1 current / 1 desired
   Conditions:
     Type            Status  Reason            Message
     ----            ------  ------            -------
     AbleToScale     True    ReadyForNewScale  recommended size matches current size
     ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from pods metric DCGM_CUSTOM_PROCESS_SM_UTIL
     ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
   Events:
     Type    Reason             Age   From                       Message
     ----    ------             ----  ----                       -------
     Normal  SuccessfulRescale  43m   horizontal-pod-autoscaler  New size: 2; reason: pods metric DCGM_CUSTOM_PROCESS_SM_UTIL above target
     Normal  SuccessfulRescale  34m   horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

   Layanan melakukan penskalaan menjadi 2 pod selama uji stres dan kembali ke 1 pod sekitar 5 menit setelah uji selesai.

Konfigurasi skalabilitas elastis terjadwal

Jenis ini menggunakan komponen ack-kubernetes-cronhpa-controller (CronHPA) untuk menyesuaikan jumlah replika Pod sesuai jadwal cron, cocok untuk pola lalu lintas yang dapat diprediksi.

1. Instal komponen CronHPA. Lihat Use CronHPA for scheduled horizontal scaling of containers.

2. Siapkan data model Qwen-7B-Chat-Int8. Lihat Deploy a vLLM inference service.

3. Terapkan layanan inferensi vLLM.

   arena serve kserve \
       --name=qwen-cronhpa \
       --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/vllm:0.4.1 \
       --gpus=1 \
       --cpu=4 \
       --memory=12Gi \
       --annotation="serving.kserve.io/autoscalerClass=external" \
       --data="llm-model:/mnt/models/Qwen-7B-Chat-Int8" \
      "python3 -m vllm.entrypoints.openai.api_server --port 8080 --trust-remote-code --served-model-name qwen --model /mnt/models/Qwen-7B-Chat-Int8 --gpu-memory-utilization 0.95 --quantization gptq --max-model-len=6144"

   Output yang diharapkan:

   inferenceservice.serving.kserve.io/qwen-cronhpa created
   INFO[0004] The Job qwen-cronhpa has been submitted successfully
   INFO[0004] You can run `arena serve get qwen-cronhpa --type kserve -n default` to check the job status

4. Uji layanan inferensi.

   # Dapatkan alamat IP Nginx Ingress.
   NGINX_INGRESS_IP=`kubectl -n kube-system get svc nginx-ingress-lb -ojsonpath='{.status.loadBalancer.ingress[0].ip}'`
   # Dapatkan hostname dari InferenceService.
   SERVICE_HOSTNAME=$(kubectl get inferenceservice qwen-cronhpa -o jsonpath='{.status.url}' | cut -d "/" -f 3)
   # Kirim permintaan uji ke endpoint chat completions vLLM.
   curl -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" \
        http://$NGINX_INGRESS_IP:80/v1/chat/completions -X POST \
        -d '{"model": "qwen", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 512, "temperature": 0.7, "top_p": 0.9, "seed": 10, "stop":["<|endoftext|>", "<|im_end|>", "<|im_start|>"]}'

   Output yang diharapkan:

   {"id":"cmpl-b7579597aa284f118718b22b83b726f8","object":"chat.completion","created":1715589652,"model":"qwen","choices":[{"index":0,"message":{"role":"assistant","content":"OK, what do you need to test?<|im_end|>"},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":10,"total_tokens":20,"completion_tokens":10}}%

5. Terapkan konfigurasi CronHPA. Contoh ini melakukan penskalaan menjadi 2 pod pada pukul 10.30 dan kembali ke 1 pod pada pukul 12.00 setiap hari.

   Perluas untuk melihat perintah konfigurasi auto scaling terjadwal

   kubectl apply -f- <<EOF
   apiVersion: autoscaling.alibabacloud.com/v1beta1
   kind: CronHorizontalPodAutoscaler
   metadata:
     name: qwen-cronhpa
     namespace: default
   spec:
      scaleTargetRef:
         apiVersion: apps/v1
         kind: Deployment
         name: qwen-cronhpa-predictor
      jobs:
      # Scale out at 10:30 every day.
      - name: "scale-up"
        schedule: "0 30 10 * * *"
        targetSize: 2
        runOnce: false
     # Scale in at 12:00 every day.
      - name: "scale-down"
        schedule: "0 0 12 * * *"
        targetSize: 1
        runOnce: false
   EOF

   Perluas untuk melihat konfigurasi penskalaan yang telah ditetapkan

   Name:         qwen-cronhpa
   Namespace:    default
   Labels:       <none>
   Annotations:  <none>
   API Version:  autoscaling.alibabacloud.com/v1beta1
   Kind:         CronHorizontalPodAutoscaler
   Metadata:
     Creation Timestamp:  2024-05-12T14:06:49Z
     Generation:          2
     Resource Version:    9205625
     UID:                 b9e72da7-262e-4***-b***-26586b7****c
   Spec:
     Jobs:
       Name:         scale-up
       Schedule:     0 30 10 * * *
       Target Size:  2
       Name:         scale-down
       Schedule:     0 0 12 * * *
       Target Size:  1
     Scale Target Ref:
       API Version:  apps/v1
       Kind:         Deployment
       Name:         qwen-cronhpa-predictor
   Status:
     Conditions:
       Job Id:           3972f7cc-bab0-482e-8cbe-7c4*******5
       Last Probe Time:  2024-05-12T14:06:49Z
       Message:
       Name:             scale-up
       Run Once:         false
       Schedule:         0 30 10 * * *
       State:            Submitted
       Target Size:      2
       Job Id:           36a04605-0233-4420-967c-ac2********6
       Last Probe Time:  2024-05-12T14:06:49Z
       Message:
       Name:             scale-down
       Run Once:         false
       Schedule:         0 0 12 * * *
       State:            Submitted
       Target Size:      1
     Scale Target Ref:
       API Version:  apps/v1
       Kind:         Deployment
       Name:         qwen-cronhpa-predictor
   Events:           <none>

   Sumber daya CronHPA memiliki dua tugas terjadwal. Deployment qwen-cronhpa-predictor melakukan penskalaan menjadi 2 pod pada pukul 10.30 dan kembali ke 1 pod pada pukul 12.00 setiap hari.

Langkah selanjutnya

Lihat Auto Scaling.