Container Service for Kubernetes:Workload scaling FAQ

If the current field in the HPA monitoring data shows unknown, it indicates that the kube-controller-manager cannot access the monitoring data source to retrieve the required data. As a result, HPA scaling fails.

Name:                                                  kubernetes-tutorial-deployment
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Mon, 10 Jun 2019 11:46:48  0530
Reference:                                             Deployment/kubernetes-tutorial-deployment
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  <unknown> / 2%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                      From                       Message
  ----     ------                   ----                     ----                       -------
  Warning  FailedGetResourceMetric  3m3s (x1009 over 4h18m)  horizontal-pod-autoscaler  unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Cause 1: The resource metrics data source is unavailable

First, you can run the kubectl top pod command to check if data is returned. If no data is returned for any pods, you can run the kubectl get apiservice command to check the status of the data source that provides resource metrics. The following code block shows a sample output.

NAME                                   SERVICE                      AVAILABLE   AGE
v1.                                    Local                        True        29h
v1.admissionregistration.k8s.io        Local                        True        29h
v1.apiextensions.k8s.io                Local                        True        29h
v1.apps                                Local                        True        29h
v1.authentication.k8s.io               Local                        True        29h
v1.authorization.k8s.io                Local                        True        29h
v1.autoscaling                         Local                        True        29h
v1.batch                               Local                        True        29h
v1.coordination.k8s.io                 Local                        True        29h
v1.monitoring.coreos.com               Local                        True        29h
v1.networking.k8s.io                   Local                        True        29h
v1.rbac.authorization.k8s.io           Local                        True        29h
v1.scheduling.k8s.io                   Local                        True        29h
v1.storage.k8s.io                      Local                        True        29h
v1alpha1.argoproj.io                   Local                        True        29h
v1alpha1.fedlearner.k8s.io             Local                        True        5h11m
v1beta1.admissionregistration.k8s.io   Local                        True        29h
v1beta1.alicloud.com                   Local                        True        29h
v1beta1.apiextensions.k8s.io           Local                        True        29h
v1beta1.apps                           Local                        True        29h
v1beta1.authentication.k8s.io          Local                        True        29h
v1beta1.authorization.k8s.io           Local                        True        29h
v1beta1.batch                          Local                        True        29h
v1beta1.certificates.k8s.io            Local                        True        29h
v1beta1.coordination.k8s.io            Local                        True        29h
v1beta1.events.k8s.io                  Local                        True        29h
v1beta1.extensions                     Local                        True        29h
...
[v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        29h]
...
v1beta1.networking.k8s.io              Local                        True        29h
v1beta1.node.k8s.io                    Local                        True        29h
v1beta1.policy                         Local                        True        29h
v1beta1.rbac.authorization.k8s.io      Local                        True        29h
v1beta1.scheduling.k8s.io              Local                        True        29h
v1beta1.storage.k8s.io                 Local                        True        29h
v1beta2.apps                           Local                        True        29h
v2beta1.autoscaling                    Local                        True        29h
v2beta2.autoscaling                    Local                        True        29h

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

If the issue persists, make sure that you have installed the metrics-server component on the Operations Management > Add-ons page of your cluster. For more information, see metrics-server.

Cause 2: Data cannot be retrieved during a rolling deployment or scale-out

By default, the collection epoch for the metrics-server component is 1 minute. After a scale-out or update is complete, metrics-server cannot retrieve monitoring data for a period of time. You can check again about 2 minutes after the scale-out or update is complete.

Cause 3: The request field is not configured

By default, HPA calculates utilization using the formula: actual usage / request. Check if the request field is included in the resource field of the pod.

Cause 4: The metric name is incorrect

Check if the metric name is correct, including its case. For example, if you specify the HPA-supported metric cpu as CPU, the current field of the monitoring data displays unknown.

During a rolling deployment, the community Controller Manager performs zero-filling for pods that do not have monitoring data. This may cause HPA to scale out too many pods. You can use the following configurations to prevent this issue.

Cluster-level configuration

# Add the following option to the startup parameters of metrics-server.
--enable-hpa-rolling-update-skipped=true  

Workload-level configuration

If you want to prevent this issue for only specific workloads, you can use one of the following two methods.

• Method 1: To temporarily pause Horizontal Pod Autoscaler (HPA) evaluation during a rolling deployment, add the following annotation to the specified workload's template.

  # Add the annotation to spec.template.metadata.annotations of the workload to temporarily pause HPA evaluation during a rolling deployment.
  HPARollingUpdateSkipped: "true"

  Show sample code

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment-basic
    labels:
      app: nginx
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: nginx
      template:
          metadata:
            labels:
              app: nginx
            annotations:
              HPARollingUpdateSkipped: "true"  # Skip HPA evaluation during a rolling deployment.
          spec:
            containers:
            - name: nginx
              image: nginx:1.7.9
              ports:
              - containerPort: 80

• Method 2: You can add the following annotation to the template of the specified workload to skip a configured ramp-up period at the beginning of an application deployment.

  # Add the annotation to spec.template.metadata.annotations of the workload to skip a configured ramp-up period at the beginning of an application deployment.
  HPAScaleUpDelay: 3m # 3m is an example. Set the time period as needed.

  Show sample code

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: nginx-deployment-basic
    labels:
      app: nginx
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: nginx
      template:
          metadata:
            labels:
              app: nginx
            annotations:
              HPAScaleUpDelay: 3m  # m stands for minutes. This indicates that HPA takes effect 3 minutes after the pod is created. Valid units are s (seconds) and m (minutes).
          spec:
            containers:
            - name: nginx
              image: nginx:1.7.9
              ports:
              - containerPort: 80

The following sample YAML file shows a deployment that has a switch added to prevent incorrect triggers.

## A Deployment is used as an example.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        HPAScaleUpDelay: 3m # m stands for minutes. This indicates that HPA takes effect 3 minutes after the pod is created. Valid units are s (seconds) and m (minutes).
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # Replace it with your exactly <image_name:tags>.
        ports:
        - containerPort: 80 

Common reasons for an inaccurate number of matched pods include the following:

• A rolling deployment is in progress.

• Other pods that do not belong to the object in `scaleTargetRef` are tagged with the same label. You can run the following command to check if other such pods exist.

  kubectl get pods -n {namespace_name} -l {value_of_status.selector_of_scale_sub-resource}