Container Service for Kubernetes:Buat kelompok node elastis dengan custom image

2.1 Verifikasi parameter node

1. Masuk ke node melalui SSH dan periksa log inisialisasi:

   cat /var/log/acs/init.log

   Output yang diharapkan:

   The node providerid is cn-zhangjiakou.i-xxxxx
   The node name is cn-zhangjiakou.192.168.66.xx
   The node labels are alibabacloud.com/nodepool-id=npf9fbxxxxxx,ack.aliyun.com=c22b1a2e122ff4fde85117de4xxxxxx,alibabacloud.com/instance-id=i-8vb7m7nt3dxxxxxxx,alibabacloud.com/external=true
   The node taints are

   Catat nilai ALIBABA_CLOUD_PROVIDER_ID, ALIBABA_CLOUD_NODE_NAME, ALIBABA_CLOUD_LABELS, dan ALIBABA_CLOUD_TAINTS. Anda akan membutuhkannya dalam parameter startup kubelet pada langkah 2.3.

2.2 Konfigurasikan lingkungan dasar

Jalankan perintah berikut untuk menyiapkan lingkungan dasar. Perintah ini melakukan tindakan berikut:

• Menginstal paket sistem yang diperlukan

• Menonaktifkan firewall, SELinux, dan swap (diperlukan oleh Kubernetes)

• Mengonfigurasi manajemen jaringan dan sinkronisasi waktu

• Mengatur batas deskriptor file sistem

# Install tool packages.
yum update -y && yum -y install  wget psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git tar curl

# Disable the firewall.
systemctl disable --now firewalld

# Disable SELinux.
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

# Disable swap partition.
sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a && sysctl -w vm.swappiness=0

# Network configuration.
systemctl disable --now NetworkManager
systemctl start network && systemctl enable network

# Time synchronization.
ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
yum install ntpdate -y
ntpdate ntp.aliyun.com

# Configure ulimit.
ulimit -SHn 65535
cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF

Setelah menyelesaikan konfigurasi lingkungan, upgrade kernel ke versi 4.18 atau lebih baru dan instal ipvsadm.

2.3 Instal containerd

Semua perintah dalam bagian ini mengonfigurasi containerd sebagai runtime kontainer, termasuk modul kernel dan jaringan CNI.

1. Unduh plugin CNI dan paket containerd:

   wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
   mkdir -p /etc/cni/net.d /opt/cni/bin
   # Extract the cni binary package
   tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/
   wget https://github.com/containerd/containerd/releases/download/v1.7.8/containerd-1.7.8-linux-amd64.tar.gz
   tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C /

2. Buat file layanan systemd containerd:

   cat > /etc/systemd/system/containerd.service <<EOF
   [Unit]
   Description=containerd container runtime
   Documentation=https://containerd.io
   After=network.target local-fs.target
   
   [Service]
   ExecStartPre=-/sbin/modprobe overlay
   ExecStart=/usr/local/bin/containerd
   Type=notify
   Delegate=yes
   KillMode=process
   Restart=always
   RestartSec=5
   LimitNPROC=infinity
   LimitCORE=infinity
   LimitNOFILE=infinity
   TasksMax=infinity
   OOMScoreAdjust=-999
   
   [Install]
   WantedBy=multi-user.target
   EOF

3. Konfigurasikan modul kernel yang diperlukan oleh containerd:

   cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
   overlay
   br_netfilter
   EOF
   systemctl restart systemd-modules-load.service

4. Konfigurasikan parameter kernel yang diperlukan oleh containerd:

   cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
   net.bridge.bridge-nf-call-iptables  = 1
   net.ipv4.ip_forward                 = 1
   net.bridge.bridge-nf-call-ip6tables = 1
   EOF
   
   # Load the kernel
   sysctl --system

5. Hasilkan dan modifikasi file konfigurasi containerd:

   mkdir -p /etc/containerd
   containerd config default | tee /etc/containerd/config.toml
   
   # Modify the containerd configuration file
   sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
   cat /etc/containerd/config.toml | grep SystemdCgroup
   sed -i "s#registry.k8s.io#m.daocloud.io/registry.k8s.io#g" /etc/containerd/config.toml
   cat /etc/containerd/config.toml | grep sandbox_image
   sed -i "s#config_path\ \=\ \"\"#config_path\ \=\ \"/etc/containerd/certs.d\"#g" /etc/containerd/config.toml
   cat /etc/containerd/config.toml | grep certs.d
   
   # Configure the accelerator
   mkdir /etc/containerd/certs.d/docker.io -pv
   cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
   server = "https://docker.io"
   [host."https://hub-mirror.c.163.com"]
     capabilities = ["pull", "resolve"]
   EOF

6. Aktifkan dan mulai containerd:

   # Reload systemd unit files after adding the service file
   systemctl daemon-reload
   
   systemctl enable --now containerd.service
   systemctl start containerd.service
   systemctl status containerd.service

7. Konfigurasikan crictl:

   wget https://mirrors.chenby.cn/https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz
   tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/
   # Generate the configuration file
   cat > /etc/crictl.yaml <<EOF
   runtime-endpoint: unix:///run/containerd/containerd.sock
   image-endpoint: unix:///run/containerd/containerd.sock
   timeout: 10
   debug: false
   EOF
   
   # Test
   systemctl restart  containerd
   crictl info

2.4 Instal kubelet dan kube-proxy

Semua perintah dalam bagian ini menginstal kubelet dan kube-proxy menggunakan file biner yang disalin dari node master, lalu mengonfigurasi kedua komponen agar otomatis dimulai saat boot.

1. Salin biner kubelet dan kube-proxy dari node master:

   scp /usr/local/bin/kube{let,-proxy} $NODEIP:/usr/local/bin/

2. Buat direktori sertifikat dan salin sertifikat dari node master:

   mkdir -p /etc/kubernetes/pki

   for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig;
     do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done

3. Konfigurasikan layanan kubelet. Ganti nilai variabel lingkungan dengan parameter kelompok node yang telah Anda catat di langkah 2.1.

   mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
   
   # Configure kubelet service on all k8s nodes
   cat > /usr/lib/systemd/system/kubelet.service << EOF
   
   [Unit]
   Description=Kubernetes Kubelet
   Documentation=https://github.com/kubernetes/kubernetes
   After=network-online.target firewalld.service containerd.service
   Wants=network-online.target
   Requires=containerd.service
   
   [Service]
   ExecStart=/usr/local/bin/kubelet \\
       --node-ip=${ALIBABA_CLOUD_NODE_NAME} \\
       --hostname-override=${ALIBABA_CLOUD_NODE_NAME} \\
       --node-labels=${ALIBABA_CLOUD_LABELS} \\
       --provider-id=${ALIBABA_CLOUD_PROVIDER_ID} \\
       --register-with-taints=${ALIBABA_CLOUD_TAINTS} \\
       --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig  \\
       --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
       --config=/etc/kubernetes/kubelet-conf.yml \\
       --container-runtime-endpoint=unix:///run/containerd/containerd.sock
   
   [Install]
   WantedBy=multi-user.target
   EOF

4. Buat file konfigurasi kubelet:

   cat > /etc/kubernetes/kubelet-conf.yml <<EOF
   apiVersion: kubelet.config.k8s.io/v1beta1
   kind: KubeletConfiguration
   address: 0.0.0.0
   port: 10250
   readOnlyPort: 10255
   authentication:
     anonymous:
       enabled: false
     webhook:
       cacheTTL: 2m0s
       enabled: true
     x509:
       clientCAFile: /etc/kubernetes/pki/ca.pem
   authorization:
     mode: Webhook
     webhook:
       cacheAuthorizedTTL: 5m0s
       cacheUnauthorizedTTL: 30s
   cgroupDriver: systemd
   cgroupsPerQOS: true
   clusterDNS:
   - 10.96.0.10
   clusterDomain: cluster.local
   containerLogMaxFiles: 5
   containerLogMaxSize: 10Mi
   contentType: application/vnd.kubernetes.protobuf
   cpuCFSQuota: true
   cpuManagerPolicy: none
   cpuManagerReconcilePeriod: 10s
   enableControllerAttachDetach: true
   enableDebuggingHandlers: true
   enforceNodeAllocatable:
   - pods
   eventBurst: 10
   eventRecordQPS: 5
   evictionHard:
     imagefs.available: 15%
     memory.available: 100Mi
     nodefs.available: 10%
     nodefs.inodesFree: 5%
   evictionPressureTransitionPeriod: 5m0s
   failSwapOn: true
   fileCheckFrequency: 20s
   hairpinMode: promiscuous-bridge
   healthzBindAddress: 127.0.0.1
   healthzPort: 10248
   httpCheckFrequency: 20s
   imageGCHighThresholdPercent: 85
   imageGCLowThresholdPercent: 80
   imageMinimumGCAge: 2m0s
   iptablesDropBit: 15
   iptablesMasqueradeBit: 14
   kubeAPIBurst: 10
   kubeAPIQPS: 5
   makeIPTablesUtilChains: true
   maxOpenFiles: 1000000
   maxPods: 110
   nodeStatusUpdateFrequency: 10s
   oomScoreAdj: -999
   podPidsLimit: -1
   registryBurst: 10
   registryPullQPS: 5
   resolvConf: /etc/resolv.conf
   rotateCertificates: true
   runtimeRequestTimeout: 2m0s
   serializeImagePulls: true
   staticPodPath: /etc/kubernetes/manifests
   streamingConnectionIdleTimeout: 4h0m0s
   syncFrequency: 1m0s
   volumeStatsAggPeriod: 1m0s
   EOF

5. Aktifkan dan mulai kubelet:

   # Reload systemd unit files after adding the service file
   systemctl daemon-reload
   
   systemctl enable --now kubelet.service
   systemctl start kubelet.service
   systemctl status kubelet.service

6. Verifikasi node telah bergabung ke kluster:

   kubectl get node

7. Salin kubeconfig kube-proxy dari node master:

   scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig

8. Buat file layanan kube-proxy:

   cat >  /usr/lib/systemd/system/kube-proxy.service << EOF
   [Unit]
   Description=Kubernetes Kube Proxy
   Documentation=https://github.com/kubernetes/kubernetes
   After=network.target
   
   [Service]
   ExecStart=/usr/local/bin/kube-proxy \\
     --config=/etc/kubernetes/kube-proxy.yaml \\
     --v=2
   Restart=always
   RestartSec=10s
   
   [Install]
   WantedBy=multi-user.target
   
   EOF

9. Buat file konfigurasi kube-proxy:

   cat > /etc/kubernetes/kube-proxy.yaml << EOF
   apiVersion: kubeproxy.config.k8s.io/v1alpha1
   bindAddress: 0.0.0.0
   clientConnection:
     acceptContentTypes: ""
     burst: 10
     contentType: application/vnd.kubernetes.protobuf
     kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
     qps: 5
   clusterCIDR: 172.16.0.0/12,fc00:2222::/112
   configSyncPeriod: 15m0s
   conntrack:
     max: null
     maxPerCore: 32768
     min: 131072
     tcpCloseWaitTimeout: 1h0m0s
     tcpEstablishedTimeout: 24h0m0s
   enableProfiling: false
   healthzBindAddress: 0.0.0.0:10256
   hostnameOverride: ""
   iptables:
     masqueradeAll: false
     masqueradeBit: 14
     minSyncPeriod: 0s
     syncPeriod: 30s
   ipvs:
     masqueradeAll: true
     minSyncPeriod: 5s
     scheduler: "rr"
     syncPeriod: 30s
   kind: KubeProxyConfiguration
   metricsBindAddress: 127.0.0.1:10249
   mode: "ipvs"
   nodePortAddresses: null
   oomScoreAdj: -999
   portRange: ""
   udpIdleTimeout: 250ms
   EOF

10. Aktifkan dan mulai kube-proxy:

     # Reload systemd unit files after adding the service file
     systemctl daemon-reload
    
     systemctl enable --now kube-proxy.service
     systemctl restart kube-proxy.service
     systemctl status kube-proxy.service

2.5 Sinkronkan status kelompok node dan ekspor image

1. Di Konsol ACK, buka Clusters. Klik nama kluster, lalu buka Nodes > Node Pools.

2. Klik Sync Node Pool dan tunggu hingga proses sinkronisasi selesai. Pastikan tidak ada node dalam status Failed.

   

3. Di Konsol ECS, buka Instances & Images > Instances.

4. Klik Instance ID, buka tab Instance Details, lalu klik Create Custom Image.

   

5. Buka Instances & Images > Images. Pastikan custom image muncul dengan Status sebagai Available.