Auto Scaling:Enable node autoscaling in self-managed Kubernetes

(Optional) Step 1: Build a Cluster Autoscaler image

Build a Cluster Autoscaler image from the source code to deploy in your Kubernetes cluster.

1. Download the source code from GitHub.

   mkdir -p $GOPATH/src/github.com/kubernetes
   cd $GOPATH/src/github.com/kubernetes
   git clone https://github.com/kubernetes/autoscaler.git
   cd autoscaler

2. Build the image.

   # Compile
   cd cluster-autoscaler && make build-arch-amd64
   # Build the image
   docker build -t cluster-autoscaler:v1.0 -f Dockerfile.amd64 .
   # Tag the image
   docker tag cluster-autoscaler:v1.0 your-image-repository-domain/cluster-autoscaler:v1.0
   # Push the image
   docker push your-image-repository-domain/cluster-autoscaler:v1.0

Step 2: Configure a scaling group

1. Create a scaling group.

   1. Log on to the Alibaba Cloud Auto Scaling Console.

   2. In the top navigation bar, select a Zone. In the left-side navigation pane, click Scaling Groups, and then click Create.

   3. On the Create by Form tab, configure the scaling group parameters and click Create. The following table shows sample configurations. For more information about how to configure a scaling group, see Create a scaling group.

      Parameter	Description	Example
      Scaling Group Name	Enter a name for the scaling group. The name must meet the format requirements.	K8s-Node-Scaling-Group
      Type	Select ECS instances. This indicates that the instances in the scaling group are ECS instances.	ECS instances
      Instance Configuration Source	With this option, you create a scaling configuration manually after the scaling group is created, instead of using a template.	Create from Scratch
      Minimum Number of Instances	The minimum number of ECS instances that a scaling group can contain. Set to 0.	0
      Maximum Number of Instances	The maximum number of ECS instances that a scaling group can contain. Set to 5.	5
      VPC	Specifies the VPC where the ECS instances of the scaling group are created.	vpc-test****-001
      vSwitch	Configuring vSwitches in multiple zones improves the reliability of scale-out activities.	vsw-test****

      Important

      After the scaling group is created, record its Zone and Scaling Group ID for later use.

2. Create a scaling configuration for the scaling group.

   1. Find the scaling group that you just created, and click View Details to go to the scaling group details page.

   2. On the Instance Configuration Source tab, click the Scaling Configurations tab, and then click Create Scaling Configuration to go to the Create Scaling Configuration page.

   3. This example uses the following sample configurations. For more information about how to create a scaling configuration, see Create a scaling configuration for ECS instances.

      Parameter	Description	Example
      Scaling Configuration Name	Enter a name for the scaling configuration. The name must meet the format requirements.	K8s-Scaling-Node-Config
      Billing Method	You can select a billing method based on your needs.	Pay-as-you-go
      Instance Configuration Mode	You can select an instance configuration mode based on your needs.	Specify Instance Type
      Select Instance Type	You can select an instance type based on your needs. Warning The following instance families are supported: Enterprise-level x86 compute instance families Enterprise-level heterogeneous compute instance families High-performance computing instance families Elastic Bare Metal Server instance families Enterprise-level Arm-based compute instance families are not supported. For more information about instance families, see Instance families.	ecs.g6a.large
      Select Image	Select a suitable image based on your needs.	Alibaba Cloud Linux

   4. Configure Network and security group.

      • Security Group: Select a security group that allows connections to the network where your Kubernetes cluster resides.

      • Assign public IPv4 address: If the API server of your Kubernetes cluster has a public IP address, select this checkbox to enable public network access for the instance.

        Warning

        If the API server of your Kubernetes cluster has a public IP address, make sure that port 6443 is open.

   5. Configure Advanced Settings > User Data by entering the following script in the User Data field to initialize the Kubernetes Worker node environment and add the Worker node to the Kubernetes cluster.

      Important

      Replace <<YOUR_MASTER_NODE_IP>> with the IP address of your Kubernetes master node.

      #!/bin/bash
      
      # Disable the firewall
      systemctl stop firewalld
      systemctl disable firewalld
      
      # Disable SELinux
      sed -i 's/enforcing/disabled/' /etc/selinux/config  # Permanent
      setenforce 0  # Temporary
      
      # Disable swap
      swapoff -a  # Temporary
      sed -ri 's/.swap./#&/' /etc/fstab    # Permanent
      
      # Pass bridged IPv4 traffic to iptables chains
      cat > /etc/sysctl.d/k8s.conf << EOF
      net.bridge.bridge-nf-call-ip6tables = 1
      net.bridge.bridge-nf-call-iptables = 1
      EOF
      sysctl --system  # Apply changes
      
      
      # Add the Kubernetes repository
      cat <<EOF > /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
      enabled=1
      gpgcheck=1
      repo_gpgcheck=1
      gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
      EOF
      
      
      # Install general-purpose packages
      yum install vim bash-completion net-tools gcc -y
      
      # Install Docker
      wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
      yum -y install docker-ce
      
      systemctl enable docker && systemctl start docker
      
      cat > /etc/docker/daemon.json << EOF
      {
        "exec-opts": ["native.cgroupdriver=systemd"]
      }
      EOF
      
      systemctl restart docker
      
      
      # Install kubeadm, kubectl, and kubelet
      yum install -y kubelet-1.23.0 kubeadm-1.23.0 kubectl-1.23.0
      
      # Start the kubelet service
      systemctl enable kubelet && systemctl start kubelet
      
      # If kubelet fails to start, use this command to troubleshoot: journalctl -xeu kubelet
      
      # Join the worker node to the cluster
      regionId=$(sed -n 's/.*"region-id": "\(.*\)".*/\1/p' /run/cloud-init/instance-data.json)
      instanceId=$(sed -n 's/.*"instance_id": "\(.*\)".*/\1/p' /run/cloud-init/instance-data.json)
      privateIpv4=$(sed -n 's/.*"private-ipv4": "\(.*\)".*/\1/p' /run/cloud-init/instance-data.json)
      
      cat > kubeadm-config.yaml << EOF
      apiVersion: kubeadm.k8s.io/v1beta2
      kind: JoinConfiguration
      discovery:
        bootstrapToken:
          token: "your-bootstrap-token"
          apiServerEndpoint: "<<YOUR_MASTER_NODE_IP>>:6443"
          caCertHashes:
          - "sha256:your-discovery-token-ca-cert-hash"
      nodeRegistration:
        name: "$regionId-$privateIpv4"
        kubeletExtraArgs:
          provider-id: "$regionId.$instanceId"
      
      EOF
      
      kubeadm join --config=kubeadm-config.yaml

      Note

      You must specify the --provider-id parameter for the worker node during a scale-out. This is handled by the script.

   6. Click Create and ensure that the scaling configuration is enabled.

3. (Optional) Verify that instances scaled out by the scaling group can join the Kubernetes cluster.

   You can manually change the minimum number of instances in the scaling group to 1 to scale out one ECS instance. Then, check whether the new ECS instance is initialized and has successfully joined your Kubernetes cluster.

Step 3: Deploy Cluster Autoscaler

1. Base64-encode the AccessKey ID and AccessKey secret of the RAM user that you prepared.

   echo -n $AccessKey-ID | base64
   echo -n $AccessKey-Secret | base64 
   echo -n $RegionId | base64

2. Create a file named deploy-ca.yaml with the following content. After modifying the specified fields, deploy the file to the kube-system namespace of your Kubernetes cluster.

   Important

   In the YAML file, make the following replacements:

   • Replace <<YOUR_ACCESS_KEY_ID>> with the Base64-encoded AccessKey ID.

   • Replace <<YOUR_ACCESS_KEY_SECRET>> with the Base64-encoded AccessKey secret.

   • Replace <<YOUR_REGION_ID>> with the Base64-encoded region ID. For more information about how to obtain a region ID, see Regions and zones.

   • Replace <<YOUR_ESS_SCALING_GROUP_ID>> with the Scaling Group ID of the scaling group you created.

   • Replace <<KUBERNETES_SERVICE_HOST>> with the API server address of your Kubernetes cluster.

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: cloud-config
   type: Opaque
   data:
     access-key-id: <<YOUR_ACCESS_KEY_ID>>
     access-key-secret: <<YOUR_ACCESS_KEY_SECRET>>
     region-id: <<YOUR_REGION_ID>>
   
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     labels:
       k8s-addon: cluster-autoscaler.addons.k8s.io
       k8s-app: cluster-autoscaler
     name: cluster-autoscaler
     namespace: kube-system
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: cluster-autoscaler
     labels:
       k8s-addon: cluster-autoscaler.addons.k8s.io
       k8s-app: cluster-autoscaler
   rules:
   - apiGroups: [""]
     resources: ["events","endpoints"]
     verbs: ["create", "patch"]
   - apiGroups: [""]
     resources: ["pods/eviction"]
     verbs: ["create"]
   - apiGroups: [""]
     resources: ["pods/status"]
     verbs: ["update"]
   - apiGroups: [""]
     resources: ["endpoints"]
     resourceNames: ["cluster-autoscaler"]
     verbs: ["get","update"]
   - apiGroups: [""]
     resources: ["nodes"]
     verbs: ["watch","list","get","update"]
   - apiGroups: [""]
     resources: ["namespaces","pods","services","replicationcontrollers","persistentvolumeclaims","persistentvolumes"]
     verbs: ["watch","list","get"]
   - apiGroups: ["extensions"]
     resources: ["replicasets","daemonsets"]
     verbs: ["watch","list","get"]
   - apiGroups: ["policy"]
     resources: ["poddisruptionbudgets"]
     verbs: ["watch","list"]
   - apiGroups: ["apps"]
     resources: ["statefulsets", "replicasets", "daemonsets"]
     verbs: ["watch","list","get"]
   - apiGroups: ["batch"]
     resources: ["jobs"]
     verbs: ["watch","list","get"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["storageclasses", "csinodes", "csidrivers", "csistoragecapacities"]
     verbs: ["watch","list","get"]
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: cluster-autoscaler
     namespace: kube-system
     labels:
       k8s-addon: cluster-autoscaler.addons.k8s.io
       k8s-app: cluster-autoscaler
   rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["create","list","watch"]
   - apiGroups: [""]
     resources: ["configmaps"]
     resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
     verbs: ["delete","get","update","watch"]
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: cluster-autoscaler
     labels:
       k8s-addon: cluster-autoscaler.addons.k8s.io
       k8s-app: cluster-autoscaler
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: cluster-autoscaler
   subjects:
     - kind: ServiceAccount
       name: cluster-autoscaler
       namespace: kube-system
   
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: cluster-autoscaler
     namespace: kube-system
     labels:
       k8s-addon: cluster-autoscaler.addons.k8s.io
       k8s-app: cluster-autoscaler
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: cluster-autoscaler
   subjects:
     - kind: ServiceAccount
       name: cluster-autoscaler
       namespace: kube-system
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: cluster-autoscaler
     name: cluster-autoscaler
     namespace: kube-system
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: cluster-autoscaler
     template:
       metadata:
         labels:
           app: cluster-autoscaler
       spec:
         dnsPolicy: "None"
         dnsConfig:
           nameservers:
             - 100.100.2.136
             - 100.100.2.138
           options:
             - name: timeout
               value: "1"
             - name: attempts
               value: "3"
         priorityClassName: system-cluster-critical
         serviceAccountName: cluster-autoscaler
         containers:
           - command:
               - ./cluster-autoscaler
               - '--v=2'
               - '--logtostderr=true'
               - '--stderrthreshold=info'
               - '--cloud-provider=alicloud'
               - '--expander=least-waste'
               - '--scan-interval=60s'
               - '--scale-down-enabled=true'
               - '--scale-down-delay-after-add=10m'
               - '--scale-down-delay-after-failure=1m'
               - '--scale-down-unready-time=2m'
               - '--ok-total-unready-count=1000'
               - '--max-empty-bulk-delete=50'
               - '--leader-elect=false'
               - '--max-node-provision-time=5m'
               - '--scale-up-from-zero=true'
               - '--daemonset-eviction-for-empty-nodes=false'
               - '--daemonset-eviction-for-occupied-nodes=false'
               - '--max-graceful-termination-sec=14400'
               - '--skip-nodes-with-system-pods=true'
               - '--skip-nodes-with-local-storage=false'
               - '--min-replica-count=0'
               - '--scale-down-unneeded-time=10m'
               - '--scale-down-utilization-threshold=0.3'
               - '--scale-down-gpu-utilization-threshold=0.3'
               - '--nodes=0:100:<<YOUR_ESS_SCALING_GROUP_ID>>'
             image: >-
               ess-cluster-autoscaler-registry.cn-hangzhou.cr.aliyuncs.com/ess-cluster-autoscaler/cluster-autoscaler:v1.7
             imagePullPolicy: Always
             name: cluster-autoscaler
             resources:
               requests:
                 cpu: 100m
                 memory: 300Mi
             securityContext:
               allowPrivilegeEscalation: true
               capabilities:
                 add:
                   - SYS_ADMIN
                 drop:
                   - ALL
             env:
             - name: ACCESS_KEY_ID
               valueFrom:
                 secretKeyRef:
                   name: cloud-config
                   key: access-key-id
             - name: ACCESS_KEY_SECRET
               valueFrom:
                 secretKeyRef:
                   name: cloud-config
                   key: access-key-secret
             - name: REGION_ID
               valueFrom:
                 secretKeyRef:
                   name: cloud-config
                   key: region-id
             - name: KUBERNETES_SERVICE_HOST
               value: "<<KUBERNETES_SERVICE_HOST>>"
             - name: KUBERNETES_SERVICE_PORT
               value: "6443"
             - name: KUBERNETES_SERVICE_PORT_HTTPS
               value: "6443"

   Note

   • Use the--scale-down-enabled parameter to control whether to enable scale-in. If scale-in is enabled, CA periodically checks the cluster status to identify nodes with resource utilization below the threshold defined by the--scale-down-utilization-threshold parameter (default is 50%).

   • By default, CA does not terminate nodes that run pods from the kube-system namespace. You can specify--skip-nodes-with-system-pods=false to override this behavior.

   • By default, CA waits 10 minutes before a scale-in operation. You can change this duration by using the--scale-down-delay-after-add parameter, for example,--scale-down-delay-after-add=5m.

   • If you use multiple scaling groups, the--expander parameter supports the following options:random,most-pods, andleast-waste.

     • random: Randomly selects a scaling group to scale out.

     • most-pods: Selects the scaling group that can schedule the most pods for scale-out.

     • least-waste: Selects the scaling group that will have the least idle CPU and memory resources after scale-out. If multiple scaling groups are tied, the system falls back to the random option.

   Run the following command to deploy CA to the Kubernetes cluster:

   kubectl apply -f deploy-ca.yaml -n kube-system

CA triggers a scale-out when pods enter a Pending state due to insufficient resources. Conversely, it triggers a scale-in when a node's resource utilization is consistently below the predefined threshold.

1. Deploy a simple nginx-demo.yaml file to verify the automatic scale-out feature. The following is a sample YAML file:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-demo
   spec:
     selector:
       matchLabels:
         app: nginx-demo
     replicas: 2
     template:
       metadata:
         labels:
           app: nginx-demo
       spec:
         containers:
           - name: nginx
             image: ess-cluster-autoscaler-registry.cn-hangzhou.cr.aliyuncs.com/ess-cluster-autoscaler/nginx-demo:v1.0
             ports:
               - containerPort: 80
                 name: http
               - containerPort: 443
                 name: https
             resources:
               requests:
                 memory: 1Gi
                 cpu: 1
               limits:
                 memory: 1Gi
                 cpu: '1'

   Run the following command to deploy the nginx-demo.yaml file:

   kubectl apply -f nginx-demo.yaml

2. Increase the replica count to create Pending pods, which will trigger a scale-out due to insufficient resources. Run the following command to increase the number of replicas:

   kubectl scale deployment nginx-demo --replicas=5

3. Wait for about one minute and check whether the scaling group has scaled out.

4. After the new instances are running, wait for about three minutes and check whether they have joined the cluster. Run the following command to view all nodes in the cluster and check whether new nodes have been added:

   kubectl get nodes