If you migrate a user-created Kubernetes cluster to an ACK cluster, we recommend that you use the default system image and default system services to create the ACK cluster. However, you can also use a custom image to create ACK clusters based on your business requirements. This topic describes how to use a custom image to create an ACK cluster.
Background information
If you migrate a user-created Kubernetes cluster to an ACK cluster, we recommend that you use the default system image for CentOS 7.6 or Alibaba Cloud Linux 2.1903 and default system services. The default system services include the operating system kernel, domain name system (DNS) service, and YUM repositories. If you want to use a custom image to create an ACK cluster, use the ack-image-builder tool to create a custom image.
Use ack-image-builder to create a custom image
The ack-image-builder tool is developed based on open source tool HashiCorp Packer. The ack-image-builder tool provides a default template and a verification script for you to create custom images.
By using ack-image-builder, you can reduce errors caused by manual operations. The ack-image-builder tool also records image changes to facilitate troubleshooting. To use the ack-image-builder tool to create a custom image for an ACK cluster, perform the following steps:
Use a custom operating system kernel
ACK requires a Linux operating system with the kernel of V3.10
or later. We recommend that you update only the RPM packages to be customized. You
must set boot parameters for the kernel.
cat scripts/updateOSKernel.sh
#! /bin/bash
VERSION_KERNEL="3.10.0-1062.4.3.el7"
yum localinstall -y http://xxx.xxx.xxx.xxx/kernel-${VERSION_KERNEL}.x86_64.rpm http://xxx.xxx.xxx.xxx/kernel-devel-${VERSION_KERNEL}.x86_64.rpm http://xxx.xxx.xxx.xxx/kernel-headers-${VERSION_KERNEL}.x86_64.rpm
grub_num=$(cat /etc/grub2.cfg |awk -F\' '$1=="menuentry " {print i " : " $2}' |grep $VERSION_KERNEL |awk -F ':' '{print $1}')
grub2-set-default $grub_num
yum update -y
command.
Customize the operating system kernel
["vm.max_map_count"]="262144"
["kernel.softlockup_panic"]="1"
["kernel.softlockup_all_cpu_backtrace"]="1"
["net.core.somaxconn"]="32768"
["net.core.rmem_max"]="16777216"
["net.core.wmem_max"]="16777216"
["net.ipv4.tcp_wmem"]="4096 12582912 16777216"
["net.ipv4.tcp_rmem"]="4096 12582912 16777216"
["net.ipv4.tcp_max_syn_backlog"]="8096"
["net.ipv4.tcp_slow_start_after_idle"]="0"
["net.core.netdev_max_backlog"]="16384"
["fs.file-max"]="2097152"
["fs.inotify.max_user_instances"]="8192"
["fs.inotify.max_user_watches"]="524288"
["fs.inotify.max_queued_events"]="16384"
["net.ipv4.ip_forward"]="1"
["net.bridge.bridge-nf-call-iptables"]="1"
["fs.may_detach_mounts"]="1"
["net.ipv4.conf.default.rp_filter"]="0"
["net.ipv4.tcp_tw_reuse"]="0"
["net.ipv4.tcp_tw_recycle"]="0"
Use a custom DNS service
- Add Alibaba Cloud name servers to the upstream name servers of the custom DNS service.
cat /etc/resolv.conf options timeout:2 attempts:3 rotate single-request-reopen ; generated by /usr/sbin/dhclient-script nameserver 100.XX.XX.136 nameserver 100.XX.XX.138
- Lock the /etc/resolve.conf file after you modify it. Otherwise, cloud-init restores the file to default settings
after ECS instances restart. You can use the following code:
cat scripts/updateDNS.sh #! /bin/bash # unlock DNS file in case it was locked chattr -i /etc/resolv.conf # Using your custom nameserver to replace xxx.xxx.xxx.xxx echo -e "nameserver xxx.xxx.xxx.xxx\nnameserver xxx.xxx.xxx.xxx" > /etc/resolv.conf # Keep resolv locked to prevent overwriting by cloudinit/NetworkManager chattr i /etc/resolv.conf
- Ensure adequate performance of the custom DNS service.
Make sure that the performance of the custom DNS service can meet the requirements if your cluster contains a large number of nodes.
Use a custom YUM repository
- Do not update all RPM packages.
Update only the RPM packages to be installed. Do not run the
yum update -y
command to update all RPM packages. - Ensure adequate performance of the YUM repository.
If you want to add a large number of worker nodes to the cluster at a time and update RPM packages based on the YUM repository, make sure that the performance of the YUM repository can meet your business requirements. You can use the following code:
cat scripts/add-yum-repo.sh #! /bin/bash cat << EOF > /etc/yum.repos.d/my.repo [base] name=CentOS-\$releasever enabled=1 failovermethod=priority baseurl=http://mirrors.cloud.aliyuncs.com/centos/\$releasever/os/\$basearch/ gpgcheck=1 gpgkey=http://mirrors.cloud.aliyuncs.com/centos/RPM-GPG-KEY-CentOS-7 EOF
Preload the container images of DaemonSet components
If you want to add more than 1,000 worker nodes to the cluster at a time, we recommend that you preload the container images of DaemonSet components before you create the custom image. This reduces the workload of pulling these container images when nodes start and improves the efficiency of cluster scale-outs.
Edit the configuration file of the custom image
Add the following configurations about provisioners
to the alicloud.json
file for creating the custom image:
"provisioners": [
{
"type": "shell",
"scripts": [
"config/default.sh",
"scripts/updateOSKernel.sh",
"scripts/updateDNS.sh",
"scripts/add-yum-repo.sh",
"scripts/prepare-images.sh",
"scripts/reboot.sh",
"scripts/verify.sh"
],
"expect_disconnect": true
}
]
config/default.sh
, scripts/reboot.sh
, and scripts/verify.sh
scripts are default scripts that you must run. Others are custom scripts.
The config/default.sh
script sets the time zone and disables swap partitions.
The scripts/verify.sh
script checks whether the custom image meets the requirements of the desired ACK
cluster.
After you edit the configuration file of the custom image, you can create the custom image and use it to create or scale out an ACK cluster.
Create an ACK cluster
We recommend that you first create a dedicated Kubernetes cluster that contains no worker nodes or a managed Kubernetes cluster that contains two worker nodes, add worker nodes that use a custom image to the cluster, and verify the result. This saves time and decreases the probability of errors.