This topic describes how to implement auto scaling based on Alibaba Cloud metrics.
PrerequisitesA Container Service for Kubernetes (ACK) cluster is created. For more information, see Create an ACK managed cluster, Create an ACK dedicated cluster, and Create an ASK cluster.
In many scenarios, you may want to scale applications based on metrics such as the HTTP request rate and the query rate of the Ingress. By default, the HPA does not support custom or external metrics. However, Kubernetes provides the external metrics API that allows you to implement auto scaling in a flexible manner.
- Log on to the ACK console and choose in the left-side navigation pane.
- On the All tab of the Marketplace page, search for and click ack-alibaba-cloud-metrics-adapter.
- In the upper-right corner of the ack-alibaba-cloud-metrics-adapter page, click Deploy.
- In the Deploy wizard, select a cluster and namespace, and then click Next.
- On the Parameters wizard page, select a chart version, configure the required parameters, and then click OK.
- On the Clusters page, click the name of a cluster and choose in the left-side navigation pane. On the Helm page, you can find that ack-alibaba-cloud-metrics-adapter is deployed in the cluster.
- Log on to the ACK console and click Clusters in the left-side navigation pane.
- On the Clusters page, click the name of a cluster and choose in the left-side navigation pane.
- On the Deployments page, click Create from YAML in the upper-right corner.
- Set Sample Template to Custom. Use the following YAML template to create a Deployment and a ClusterIP type Service. Then, click Create.
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment-basic labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: app: nginx type: ClusterIP
- In the left-side navigation pane, click Ingresses page, click Create in the upper-right corner. . On the
- In the Create dialog box, set the parameters and click Create. After you create an Ingress, the Ingresses page appears.
- On the Ingresses page, find the newly created Ingress and click Details in the Actions column to view information about the Ingress.
- Configure HPA.
- Click in the upper-left corner to return to the Clusters page.
- In the left-side navigation pane of the ACK console, choose .
- On the Templates page, find HPA and click Create Application to deploy an application with the following YAML template:
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: ingress-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-deployment-basic minReplicas: 2 maxReplicas: 10 metrics: - type: External external: metric: name: sls_ingress_qps selector: matchLabels: sls.project: "***" # Replace with the Log Service project that you want to use. sls.logstore: "nginx-ingress" sls.ingress.route: "default-nginx-80" target: type: AverageValue averageValue: 10 - type: External external: metric: name: sls_ingress_latency_p9999 selector: matchLabels: # default ingress log project is k8s-log-clusterId sls.project: "***" # default ingress logstre is nginx-ingress sls.logstore: "nginx-ingress" # namespace-svc-port sls.ingress.route: "default-nginx-80" # sls vpc endpoint, default true # sls.internal.endpoint:true target: type: Value # sls_ingress_latency_p9999 > 10ms value: 10The following table describes the parameters that are used to configure HPA.
Parameter Description sls.ingress.route
Set the value in the
<namespace>-<svc>-<port>format, for example, default-nginx-80. This parameter is required.Note
<namespace>specifies the namespace to which the Ingress belongs.
<svc>specifies the name of the Service that you selected when you created the Ingress.
<port>specifies the port of the Service.
sls.logstore The name of the Logstore in Log Service. This parameter is required. sls.project The name of the Log Service project. This parameter is required. sls.internal.endpoint Specifies whether to access Log Service over the internal network. Default value: true. If you set the value to true, you access Log Service over the internal network. If you set the value to false, you access Log Service over the Internet.NoteIn this example, HPA triggers scaling activities based on the sls_ingress_qps and sls_ingress_latency_p9999 metrics. In the target sections, each metric has a different type value:
- The type value of the sls_ingress_qps metric is set to AverageValue. This indicates that the metric value is the result of dividing the total QPS by the number of pods.
- The type value of the sls_ingress_latency_p9999 metric is set to Value. This indicates that the latency is not divided by the number of pods.
- On the HPA page, click Create on the right side of the editor.
- After HPA is configured, run the following script to perform a stress test:
#!/bin/bash ##Use Apache Benchmark to send requests to the Service exposed by the Ingress. The test lasts 300 seconds and 10 concurrent requests are sent per second. ab -t 300 -c 10 <The domain name of the Ingress>
- Check whether HPA works as expected.
- In the left-side navigation pane of the ACK console, click Clusters. On the Clusters page, find the cluster that you want to manage and choose in the Actions column.
- Run the following command to check the status of HPA:
kubectl get hpa ingress-hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAPS AGE ingress-hpa Depolyment/nginx-deployment-basic 21/10 (avg) 2 10 10 7m49s
If the value of REPLICAS is the same as the value of MAXPODS, it indicates that HPA scaled out the number of pods as expected.
What do I do if the TARGETS column shows <unknow> after I run the
kubectl get hpacommand?Perform the following operations to troubleshoot the issue:
- Run the
kubectl describe hpa <hpa_name>command to check why HPA does not function as normal.
- If the value of AbleToScale is False in the Conditions field, check whether the Deployment is created as normal.
- If the value of ScalingActive is False in the Conditions field, proceed to the next step.
- Run the
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/"command. If Error from server (NotFound): the server could not find the requested resource is returned, verify the status of alibaba-cloud-metrics-adapter.
If the status of alibaba-cloud-metrics-adapter is normal, check whether the HPA metrics are relevant to the Ingress. If the metrics are relevant to the Ingress, make sure that you deploy the Log Service component before ack-alibaba-cloud-metrics-adapter is deployed. For more information see Analyze and monitor the access log of nginx-ingress.
- Make sure that the values of the HPA metrics are valid. The value of sls.ingress.route must be in the
- namespace specifies the namespace to which the Ingress belongs.
- svc specifies the name of the Service that you selected when you created the Ingress.
- port specifies the port of the Service.
- Run the
Where can I find the metrics that are supported by HPA?For more information about the metrics that are supported by HPA, see Alibaba Cloud metrics adapter. The following table describes the commonly used metrics.
Metric Description Additional parameter sls_ingress_qps The number of requests that the Ingress can process per second based on a specific routing rule. sls.ingress.route sls_alb_ingress_qps The number of requests that the ALB Ingress can process per second based on a specific routing rule. sls.ingress.route sls_ingress_latency_avg The average latency of all requests. sls.ingress.route sls_ingress_latency_p50 The maximum latency for the fastest 50% of all requests. sls.ingress.route sls_ingress_latency_p95 The maximum latency for the fastest 95% of all requests. sls.ingress.route sls_ingress_latency_p99 The maximum latency for the fastest 99% of all requests. sls.ingress.route sls_ingress_latency_p9999 The maximum latency for the fastest 99.99% of all requests. sls.ingress.route sls_ingress_inflow The inbound bandwidth of the Ingress. sls.ingress.route
How do I collect NGINX Ingress logs in a custom format?In this topic, horizontal pod autoscaling is performed based on the Ingress metrics that are collected by Log Service. You must configure Log Service to collect NGINX Ingress logs.
- When you create an ACK cluster, Log Service is enabled for the cluster by default. If you use the default log collection settings, you can view the log analysis reports and real-time status of NGINX Ingresses in the Log Service console after you create the cluster.
- If you disable Log Service when you create an ACK cluster, you cannot perform horizontal pod autoscaling based on the Ingress metrics that are collected by Log Service. You must enable Log Service for the cluster before you can use this feature. For more information, see Analyze and monitor the access log of nginx-ingress.
- The AliyunLogConfig that is generated when you enable Log Service for the cluster for the first time applies only to the default log format that ACK defines for the Ingress controller. If you have changed the log format, you must modify the
processor_regexsettings in the AliyunLogConfig. For more information, see Use CRDs to collect container logs in DaemonSet mode.