Alibaba Cloud Container Service for Kubernetes provides the vertical pod autoscaling feature by deploying vpa (vertical-pod-autoscaler) to ACK clsuter.

Prerequisites

  • ACK kubernetes version should be 1.12 or later.
  • Kubectl should be connected to the cluster you want to install VPA in.
  • If you already have another version of VPA installed in your cluster, you have to tear down the existing installation first.

Background information

Vertical Pod Autoscaler (VPA) frees the users from necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it will set the requests automatically based on usage and thus allow proper scheduling onto nodes so that appropriate resource amount is available for each pod. It will also maintain ratios between limits and requests that were specified in initial containers configuration. For more information, see repo in github.

Note

  • Updating running pods is an experimental feature of VPA. Whenever VPA updates the pod resources the pod is recreated, which causes all running containers to be restarted. The pod may be recreated on a different node.
  • *VPA does not evict pods which are not run under a controller. For such pods Auto mode is currently equivalent to Initial.
  • Vertical Pod Autoscaler should not be used with the Horizontal Pod Autoscaler (HPA) on CPU or memory at this moment. However, you can use VPA with HPA on custom and external metrics.
  • The VPA admission controller is an admission webhook. If you add other admission webhooks to you cluster, it is important to analyze how they interact and whether they may conflict with each other. The order of admission controllers is defined by a flag on APIserver.
  • VPA reacts to most out-of-memory events, but not in all situations.
  • VPA performance has not been tested in large clusters.
  • VPA recommendation might exceed available resources (e.g. Node size, available size, available quota) and cause pods to go pending. This can be partly addressed by using VPA together with Cluster Autoscaler.
  • Multiple VPA resources matching the same pod have undefined behavior.

Install vertical-pod-autoscaler

  1. Install CRD of vertical-pod-autoscaler.
    Execute command kubectl apply -f crd.yaml and the content of crd.yaml is below.
  2. Install vertical-pod-autoscaler components.
    1. Install component admission-controller.
      Before install admission-controller. you need to generate certs beform. Please use this script to generate webhook certs.
      ---
      apiVersion: apiextensions.k8s.io/v1beta1
      kind: CustomResourceDefinition
      metadata:
        name: verticalpodautoscalers.autoscaling.k8s.io
      spec:
        group: autoscaling.k8s.io
        scope: Namespaced
        names:
          plural: verticalpodautoscalers
          singular: verticalpodautoscaler
          kind: VerticalPodAutoscaler
          shortNames:
            - vpa
        version: v1beta1
        versions:
          - name: v1beta1
            served: true
            storage: false
          - name: v1beta2
            served: true
            storage: true
        validation:
          # openAPIV3Schema is the schema for validating custom objects.
          openAPIV3Schema:
            type: object
            properties:
              spec:
                type: object
                required: []
                properties:
                  targetRef:
                    type: object
                  updatePolicy:
                    type: object
                    properties:
                      updateMode:
                        type: string
                  resourcePolicy:
                    type: object
                    properties:
                      containerPolicies:
                        type: array
                        items:
                          type: object
      ---
      apiVersion: apiextensions.k8s.io/v1beta1
      kind: CustomResourceDefinition
      metadata:
        name: verticalpodautoscalercheckpoints.autoscaling.k8s.io
      spec:
        group: autoscaling.k8s.io
        scope: Namespaced
        names:
          plural: verticalpodautoscalercheckpoints
          singular: verticalpodautoscalercheckpoint
          kind: VerticalPodAutoscalerCheckpoint
          shortNames:
            - vpacheckpoint
        version: v1beta1
        versions:
          - name: v1beta1
            served: true
            storage: false
          - name: v1beta2
            served: true
            storage: true
    2. Install component recommander.
      ```
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: vpa-recommender
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: vpa-recommender
        template:
          metadata:
            labels:
              app: vpa-recommender
          spec:
            serviceAccountName: admin
            containers:
            - name: recommender
              image: registry.cn-hangzhou.aliyuncs.com/acs/vpa-recommender:0.7.0
              imagePullPolicy: Always
              resources:
                limits:
                  cpu: 200m
                  memory: 1000Mi
                requests:
                  cpu: 50m
                  memory: 500Mi
              ports:
              - containerPort: 8080
      ```
    3. Install component updater.
      ```
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: vpa-updater
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: vpa-updater
        template:
          metadata:
            labels:
              app: vpa-updater
          spec:
            serviceAccountName: admin
            containers:
              - name: updater
                image: registry.cn-hangzhou.aliyuncs.com/acs/vpa-updater:0.7.0
                imagePullPolicy: Always
                resources:
                  limits:
                    cpu: 200m
                    memory: 1000Mi
                  requests:
                    cpu: 50m
                    memory: 500Mi
                ports:
                  - containerPort: 8080
      ```
  3. Verify installation.

    Create nginx-deployment-basic deployment and nginx-deployment-basic-vpa. Switch updateMode option off and leave the resources (requests and limits) empty.

    ```
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.7.9
            ports:
            - containerPort: 80
    ---
    apiVersion: autoscaling.k8s.io/v1
    kind: VerticalPodAutoscaler
    metadata:
      name: nginx-deployment-basic-vpa
    spec:
      targetRef:
        apiVersion: "apps/v1"
        kind:       Deployment
        name:       nginx-deployment-basic
      updatePolicy:
        updateMode: "Off"
    ```
    Few moments later. `kubectl describe vpa nginx-deployment-basic-vpa`, You can found the recommendations for CPU and memory requests like below.
    ```
    recommendation:
        containerRecommendations:
        - containerName: nginx
          lowerBound:
            cpu: 50m
            memory: 300144k
          target:
            cpu: 50m
            memory: 300144k
          upperBound:
            cpu: 8031m
            memory: 800000k
    ```

Now you can use the recommendations to update deployment and VPA could help you to do continuous optimization iteration. For more information, see github and join us on sig-autoscaling at slack.