Alibaba Cloud Container Service for Kubernetes provides the vertical pod autoscaling feature by deploying vpa (vertical-pod-autoscaler) to ACK clsuter.

Prerequisites

  • ACK kubernetes version should be 1.12 or later.
  • Kubectl should be connected to the cluster you want to install VPA in.
  • If you already have another version of VPA installed in your cluster, you have to tear down the existing installation first.

Background information

Vertical Pod Autoscaler (VPA) frees the users from necessity of setting up-to-date resource limits and requests for the containers in their pods. When configured, it will set the requests automatically based on usage and thus allow proper scheduling onto nodes so that appropriate resource amount is available for each pod. It will also maintain ratios between limits and requests that were specified in initial containers configuration. For more information, see repo in github.

Note

  • Updating running pods is an experimental feature of VPA. Whenever VPA updates the pod resources the pod is recreated, which causes all running containers to be restarted. The pod may be recreated on a different node.
  • *VPA does not evict pods which are not run under a controller. For such pods Auto mode is currently equivalent to Initial.
  • Vertical Pod Autoscaler should not be used with the Horizontal Pod Autoscaler (HPA) on CPU or memory at this moment. However, you can use VPA with HPA on custom and external metrics.
  • The VPA admission controller is an admission webhook. If you add other admission webhooks to you cluster, it is important to analyze how they interact and whether they may conflict with each other. The order of admission controllers is defined by a flag on APIserver.
  • VPA reacts to most out-of-memory events, but not in all situations.
  • VPA performance has not been tested in large clusters.
  • VPA recommendation might exceed available resources (e.g. Node size, available size, available quota) and cause pods to go pending. This can be partly addressed by using VPA together with Cluster Autoscaler.
  • Multiple VPA resources matching the same pod have undefined behavior.

Install vertical-pod-autoscaler

  1. Install CRD of vertical-pod-autoscaler.
    Execute command kubectl apply -f crd.yaml and the content of crd.yaml is below.
    ```
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      name: verticalpodautoscalers.autoscaling.k8s.io
      annotations:
        helm.sh/hook: crd-install
        helm.sh/hook-delete-policy: before-hook-creation
    spec:
      group: autoscaling.k8s.io
      scope: Namespaced
      names:
        plural: verticalpodautoscalers
        singular: verticalpodautoscaler
        kind: VerticalPodAutoscaler
        shortNames:
          - vpa
      version: v1beta1
      versions:
        - name: v1beta1
          served: false
          storage: false
        - name: v1beta2
          served: true
          storage: true
        - name: v1
          served: true
          storage: false
      validation:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: []
              properties:
                targetRef:
                  type: object
                updatePolicy:
                  type: object
                  properties:
                    updateMode:
                      type: string
                resourcePolicy:
                  type: object
                  properties:
                    containerPolicies:
                      type: array
                      items:
                        type: object
    ---
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      name: verticalpodautoscalercheckpoints.autoscaling.k8s.io
      annotations:
        helm.sh/hook: crd-install
        helm.sh/hook-delete-policy: before-hook-creation
    spec:
      group: autoscaling.k8s.io
      scope: Namespaced
      names:
        plural: verticalpodautoscalercheckpoints
        singular: verticalpodautoscalercheckpoint
        kind: VerticalPodAutoscalerCheckpoint
        shortNames:
          - vpacheckpoint
      version: v1beta1
      versions:
        - name: v1beta1
          served: false
          storage: false
        - name: v1beta2
          served: true
          storage: true
        - name: v1
          served: true
          storage: false
    ```
  2. Install vertical-pod-autoscaler components.
    1. Install component admission-controller.

      Before install admission-controller. you need to generate certs beform. Please use this script to generate webhook certs.

      ```
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: vpa-admission-controller
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: vpa-admission-controller
        template:
          metadata:
            labels:
              app: vpa-admission-controller
          spec:
            serviceAccountName: admin
            containers:
              - name: admission-controller
                image: registry.cn-hangzhou.aliyuncs.com/acs/vpa-admission-controller:0.7.0
                imagePullPolicy: Always
                env:
                  - name: NAMESPACE
                    valueFrom:
                      fieldRef:
                        fieldPath: metadata.namespace
                volumeMounts:
                  - name: tls-certs
                    mountPath: "/etc/tls-certs"
                    readOnly: true
                resources:
                  limits:
                    cpu: 200m
                    memory: 500Mi
                  requests:
                    cpu: 50m
                    memory: 200Mi
                ports:
                  - containerPort: 8000
            volumes:
              - name: tls-certs
                secret:
                  secretName: vpa-tls-certs
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: vpa-webhook
        namespace: kube-system
      spec:
        ports:
          - port: 443
            targetPort: 8000
        selector:
          app: vpa-admission-controller
      ```
    2. Install component recommander.
      ```
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: vpa-recommender
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: vpa-recommender
        template:
          metadata:
            labels:
              app: vpa-recommender
          spec:
            serviceAccountName: admin
            containers:
            - name: recommender
              image: registry.cn-hangzhou.aliyuncs.com/acs/vpa-recommender:0.7.0
              imagePullPolicy: Always
              resources:
                limits:
                  cpu: 200m
                  memory: 1000Mi
                requests:
                  cpu: 50m
                  memory: 500Mi
              ports:
              - containerPort: 8080
      ```
    3. Install component updater.
      ```
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: vpa-updater
        namespace: kube-system
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: vpa-updater
        template:
          metadata:
            labels:
              app: vpa-updater
          spec:
            serviceAccountName: admin
            containers:
              - name: updater
                image: registry.cn-hangzhou.aliyuncs.com/acs/vpa-updater:0.7.0
                imagePullPolicy: Always
                resources:
                  limits:
                    cpu: 200m
                    memory: 1000Mi
                  requests:
                    cpu: 50m
                    memory: 500Mi
                ports:
                  - containerPort: 8080
      ```
  3. Verify installation.

    Create nginx-deployment-basic deployment and nginx-deployment-basic-vpa. Switch updateMode option off and leave the resources (requests and limits) empty.

    ```
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment-basic
      labels:
        app: nginx
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.7.9
            ports:
            - containerPort: 80
    ---
    apiVersion: autoscaling.k8s.io/v1
    kind: VerticalPodAutoscaler
    metadata:
      name: nginx-deployment-basic-vpa
    spec:
      targetRef:
        apiVersion: "apps/v1"
        kind:       Deployment
        name:       nginx-deployment-basic
      updatePolicy:
        updateMode: "Off"
    ```
    Few moments later. `kubectl describe vpa nginx-deployment-basic-vpa`, You can found the recommendations for CPU and memory requests like below.
    ```
    recommendation:
        containerRecommendations:
        - containerName: nginx
          lowerBound:
            cpu: 50m
            memory: 300144k
          target:
            cpu: 50m
            memory: 300144k
          upperBound:
            cpu: 8031m
            memory: 800000k
    ```

Now you can use the recommendations to update deployment and VPA could help you to do continuous optimization iteration. For more information, see github and join us on sig-autoscaling at slack.