Container Service for Kubernetes:FAQ about auto scaling

If the current field of the HPA metrics shows unknown, it indicates that kube-controller-manager cannot access monitored data sources to fetch resource metrics.

Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48  0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Possible causes:

• Cause 1: The data sources from which resource metrics are collected are unavailable. Run the kubectl top pod command to check whether the metric data of monitored pods is returned. If no metric data is returned, run the kubectl get apiservice command to check whether the metrics-server component is available. The following output shows an example of the returned data:

  NAME                                   SERVICE                      AVAILABLE   AGE
  v1.                                    Local                        True        29h
  v1.admissionregistration.k8s.io        Local                        True        29h
  v1.apiextensions.k8s.io                Local                        True        29h
  v1.apps                                Local                        True        29h
  v1.authentication.k8s.io               Local                        True        29h
  v1.authorization.k8s.io                Local                        True        29h
  v1.autoscaling                         Local                        True        29h
  v1.batch                               Local                        True        29h
  v1.coordination.k8s.io                 Local                        True        29h
  v1.monitoring.coreos.com               Local                        True        29h
  v1.networking.k8s.io                   Local                        True        29h
  v1.rbac.authorization.k8s.io           Local                        True        29h
  v1.scheduling.k8s.io                   Local                        True        29h
  v1.storage.k8s.io                      Local                        True        29h
  v1alpha1.argoproj.io                   Local                        True        29h
  v1alpha1.fedlearner.k8s.io             Local                        True        5h11m
  v1beta1.admissionregistration.k8s.io   Local                        True        29h
  v1beta1.alicloud.com                   Local                        True        29h
  v1beta1.apiextensions.k8s.io           Local                        True        29h
  v1beta1.apps                           Local                        True        29h
  v1beta1.authentication.k8s.io          Local                        True        29h
  v1beta1.authorization.k8s.io           Local                        True        29h
  v1beta1.batch                          Local                        True        29h
  v1beta1.certificates.k8s.io            Local                        True        29h
  v1beta1.coordination.k8s.io            Local                        True        29h
  v1beta1.events.k8s.io                  Local                        True        29h
  v1beta1.extensions                     Local                        True        29h
  ...
  [v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        29h]
  ...
  v1beta1.networking.k8s.io              Local                        True        29h
  v1beta1.node.k8s.io                    Local                        True        29h
  v1beta1.policy                         Local                        True        29h
  v1beta1.rbac.authorization.k8s.io      Local                        True        29h
  v1beta1.scheduling.k8s.io              Local                        True        29h
  v1beta1.storage.k8s.io                 Local                        True        29h
  v1beta2.apps                           Local                        True        29h
  v2beta1.autoscaling                    Local                        True        29h
  v2beta2.autoscaling                    Local                        True        29h

  If the apiservice for v1beta1.metrics.k8s.io is not kube-system/metrics-server, check whether metrics-server is overwritten by Prometheus Operator. If metrics-server is overwritten by Prometheus Operator, use the following YAML template to redeploy metrics-server:

  apiVersion: apiregistration.k8s.io/v1beta1
  kind: APIService
  metadata:
    name: v1beta1.metrics.k8s.io
  spec:
    service:
      name: metrics-server
      namespace: kube-system
    group: metrics.k8s.io
    version: v1beta1
    insecureSkipTLSVerify: true
    groupPriorityMinimum: 100
    versionPriority: 100

  If no error is found after you have performed the preceding checks, see metrics-server.

• Cause 2: Metrics cannot be collected during a rolling update or scale-out activity.

  By default, metrics-server collects metrics at intervals of 1 minute. However, metrics-server must wait a few minutes before it can collect metrics after a rolling update or scale-out activity. We recommend that you query metrics 2 minutes after a rolling update or scale-out activity is complete.

• Cause 3: The request field is not specified for the pod.

  By default, HPA obtains the CPU or memory usage of the pod by calculating the value of used resources/requested resources. If the requested resources are not specified in the pod configurations, HPA cannot calculate the resource usage. Therefore, you must ensure that the request field is specified in the resource parameter of the pod configurations.

During a rolling update, kube-controller-manager performs zero filling on pods whose monitoring data cannot be collected. This may cause HPA to add an excessive number of pods. You can perform the following steps to fix this issue.

• Fix this issue for all workloads in the cluster:

  To fix this issue, we recommend that you update the metrics-server component provided by ACK to the latest version and add the following configuration to the startup settings of metrics-server:

  The following configuration takes effect on all workloads in the cluster.

  ## Add the following configuration to the startup settings of metrics-server. 
  --enable-hpa-rolling-update-skipped=true  

• Fix this issue for specified workloads. You can use one of the following methods to fix this issue for specified workloads:

  • Method 1: Add the following annotation to the template of a workload to skip HPA during rolling updates.

    ## Add the following annotation to the spec.template.metadata.annotations parameter of the workload configuration to skip HPA during rolling updates. 
    HPARollingUpdateSkipped: "true"

    Example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
        template:
            metadata:
              labels:
                app: nginx
              annotations:
                HPARollingUpdateSkipped: "true"  # Skip HPA during rolling updates. 
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80

  • Method 2: Add the following annotation to the template of a workload to skip the warm-up period before rolling updates.

    ## Add the following annotation to the spec.template.metadata.annotations parameter of the workload configuration to skip the warm-up period before rolling updates. 
    HPAScaleUpDelay: 3m # You can change the value based on your business requirements.

    Example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
        template:
            metadata:
              labels:
                app: nginx
              annotations:
                HPAScaleUpDelay: 3m  # This setting indicates that HPA takes effect 3 minutes after the pods are created. Valid units: s and m. s indicates seconds and m indicates minutes. 
            spec:
              containers:
              - name: nginx
                image: nginx:1.7.9
                ports:
                - containerPort: 80

The following YAML template provides the sample pod configurations that prevent HPA from triggering scaling activities in this scenario.

## A Deployment is used in this example.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        HPAScaleUpDelay: 3m # This setting indicates that HPA takes effect 3 minutes after the pods are created. Valid units: s and m. s indicates seconds and m indicates minutes. 
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # Replace it with your exactly <image_name:tags>.
        ports:
        - containerPort: 80 

Check whether the following situations exist:

• The instance types in the scaling group cannot fulfill the resource request from pods. Some resources provided by the specified ECS instance type are reserved or occupied for the following purposes:

  • Resources are used for virtualization or occupied by the OS during instance creation. For more information about the supported SDKs, see Why does a purchased instance have a memory size different from that defined in the instance type?.

  • Resources are occupied to run components such as the kubelet, kube-proxy, Terway, and container runtime. For more information about the supported SDKs, see Resource reservation policy.

  • By default, system components are installed for each node. Therefore, the requested pod resources must be less than the resource capacity of the instance type.

• Cross-zone scale-out activities cannot be triggered for pods that have limits on zones.

• The RAM role does not have the permissions to manage the Kubernetes cluster. You must configure RAM roles for each Kubernetes cluster that is involved in the scale-out activity. For more information about the authorization, see Step 2: Perform authorization.

• The following issues occur when you activate Auto Scaling:

  • The instance fails to be added to the cluster and a timeout error occurs.

  • The node is not ready and a timeout error occurs.

  To ensure that nodes can be accurately scaled, the auto scaling component does not perform any scaling activities before it fixes the abnormal nodes.

Check whether the following situations exist:

• The requested resource threshold of each pod is higher than the specified scale-in threshold.

• Pods that belong to the kube-system namespace are running on the node.

• A scheduling policy forces the pods to run on the current node. Therefore, the pods cannot be scheduled to other nodes.

• PodDisruptionBudget is set for the pods on the node and the minimum value of PodDisruptionBudget is reached.

You can configure a pod to allow Cluster Autoscaler to remove the node that hosts the pod or prevent Cluster Autoscaler from removing the node that hosts the pod.

• To configure a pod to prevent Cluster Autoscaler from removing the node that hosts the pod, add the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to the pod configuration.

• To configure a pod to allow Cluster Autoscaler to remove the node that hosts the pod, add the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" to the pod configuration.

The number of matching pods may be inaccurate due to the following reasons:

• A rolling update is in progress.

• Pods that do not belong to the object specified by the scaleTargetRef parameter have the label specified in the Selector field in the status section of the scale subresource.

To prevent a node from being removed by Cluster Autoscaler, add the annotation "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" to the node configurations. Run the following command to add the annotation:

kubectlannotatenode<nodename>cluster-autoscaler.kubernetes.io/scale-down-disabled=true