Container Service for Kubernetes:FAQ about nodes and node pools

• How do I manually update the kernel of GPU-accelerated nodes in a cluster?

• How do I resolve the issue that no container is started on a GPU-accelerated node?

• FAQ about adding nodes to a cluster

• How do I fix the "drain-node job execute timeout" error that occurs when I remove a node?

• How do I change the hostname of a worker node in an ACK cluster?

• How do I change the operating system for a node pool?

• What are the differences between node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter?

• How do I add existing nodes to a cluster?

• How do I use preemptible instances in a node pool?

• How do I change the maximum number of pods supported by a node?

How do I manually update the kernel version of GPU-accelerated nodes in a cluster?

1. Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

2. Set the GPU-accelerated node that you want to manage to the unschedulable state. In this example, the node cn-beijing.i-2ze19qyi8votgjz12345 is used.

   kubectl cordon cn-beijing.i-2ze19qyi8votgjz12345
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 already cordoned

3. Migrate the pods on the GPU-accelerated node to other nodes.

   kubectl drain cn-beijing.i-2ze19qyi8votgjz12345 --grace-period=120 --ignore-daemonsets=true
   
   node/cn-beijing.i-2ze19qyi8votgjz12345 cordoned
   WARNING: Ignoring DaemonSet-managed pods: flexvolume-9scb4, kube-flannel-ds-r2qmh, kube-proxy-worker-l62sf, logtail-ds-f9vbg
   pod/nginx-ingress-controller-78d847fb96-5fkkw evicted

4. Uninstall the existing nvidia-driver.

   Note

   In this example, the uninstalled driver version is 384.111. If your driver version is not 384.111, download the installation package of your driver from the official NVIDIA website and update the driver to 384.111 first.

   1. Log on to the GPU-accelerated node and run the nvidia-smi command to check the driver version.

      sudo nvidia-smi -a | grep 'Driver Version'
      Driver Version                      : 384.111

   2. Download the driver installation package.

      sudo cd /tmp/
      sudo curl -O https://cn.download.nvidia.cn/tesla/384.111/NVIDIA-Linux-x86_64-384.111.run

      Note

      The installation package is required for uninstalling the NVIDIA driver.

   3. Uninstall the driver.

      sudo chmod u+x NVIDIA-Linux-x86_64-384.111.run
      sudo sh ./NVIDIA-Linux-x86_64-384.111.run --uninstall -a -s -q

5. Update the kernel.

   Update the kernel version based on your business requirements.

6. Restart the GPU-accelerated node.

   sudo reboot

7. Log on to the GPU node and run the following command to install the kernel-devel package.

   sudo yum install -y kernel-devel-$(uname -r)

8. Go to the official NVIDIA website to download the required driver and install it on the GPU-accelerated node. In this example, the driver version 410.79 is used.

   sudo cd /tmp/
   sudo curl -O https://cn.download.nvidia.cn/tesla/410.79/NVIDIA-Linux-x86_64-410.79.run
   sudo chmod u+x NVIDIA-Linux-x86_64-410.79.run
   sudo sh ./NVIDIA-Linux-x86_64-410.79.run -a -s -q
   
   warm up GPU
   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

9. Make sure that the /etc/rc.d/rc.local file includes the following configurations. Otherwise, add the following configurations to the file.

   sudo nvidia-smi -pm 1 || true
   sudo nvidia-smi -acp 0 || true
   sudo nvidia-smi --auto-boost-default=0 || true
   sudo nvidia-smi --auto-boost-permission=0 || true
   sudo nvidia-modprobe -u -c=0 -m || true

10. Restart kubelet and Docker.

    sudo service kubelet stop
    sudo service docker restart
    sudo service kubelet start

11. Set the GPU-accelerated node to schedulable.

    kubectl uncordon cn-beijing.i-2ze19qyi8votgjz12345
    
    node/cn-beijing.i-2ze19qyi8votgjz12345 already uncordoned

12. Run the following command in the nvidia-device-plugin container to check the version of the driver installed on the GPU-accelerated node.

    kubectl exec -n kube-system -t nvidia-device-plugin-cn-beijing.i-2ze19qyi8votgjz12345 nvidia-smi
    Thu Jan 17 00:33:27 2019
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: N/A      |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  On   | 00000000:00:09.0 Off |                    0 |
    | N/A   27C    P0    28W / 250W |      0MiB / 16280MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID   Type   Process name                             Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

    Note

    If no container is launched on the GPU-accelerated node after you run the docker ps command, see What do I do if no container is launched on a GPU-accelerated node?

What do I do if no container is launched on a GPU-accelerated node?

sudo service kubelet stop
Redirecting to /bin/systemctl stop kubelet.service
sudo service docker stop
Redirecting to /bin/systemctl stop docker.service
sudo service docker start
Redirecting to /bin/systemctl start docker.service
sudo service kubelet start
Redirecting to /bin/systemctl start kubelet.service

sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

sudo docker info | grep -i cgroup
Cgroup Driver: cgroupfs

To resolve the issue, perform the following steps:

1. Create a copy of /etc/docker/daemon.json. Then, run the following commands to update /etc/docker/daemon.json.

   sudo cat >/etc/docker/daemon.json <<-EOF
   {
       "default-runtime": "nvidia",
       "runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       },
       "exec-opts": ["native.cgroupdriver=systemd"],
       "log-driver": "json-file",
       "log-opts": {
           "max-size": "100m",
           "max-file": "10"
       },
       "oom-score-adjust": -1000,
       "storage-driver": "overlay2",
       "storage-opts":["overlay2.override_kernel_check=true"],
       "live-restore": true
   }
   EOF

2. Run the following commands to restart the Docker runtime and kubelet:

   sudo service kubelet stop
   Redirecting to /bin/systemctl stop kubelet.service
   sudo service docker restart
   Redirecting to /bin/systemctl restart docker.service
   sudo service kubelet start
   Redirecting to /bin/systemctl start kubelet.service

3. Run the following command to check whether the cgroup driver is set to systemd.

   sudo docker info | grep -i cgroup
   Cgroup Driver: systemd

How do I change the hostname of a worker node in an ACK cluster?

1. Remove the worker node.

   1. Log on to the ACK console.

   2. In the left-side navigation pane of the ACK console, click Clusters.

   3. In the left-side navigation pane of the details page, choose Nodes > Nodes.

   4. On the Nodes page, find the worker node that you want to remove and choose More > Remove in the Actions column.

   5. In the dialog box that appears, select I understand the above information and want to remove the node(s). and click OK.

2. Add the worker node to the node pool again. For more information, see Manually add ECS instances.

   Then, the worker node is renamed based on the new node naming rule of the node pool.

How do I change the operating system for a node pool?

The method used to change the operating system for a node pool is similar to that used to update a node pool. To change the operating system for a node pool, perform the following steps:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, click the name of the cluster that you want to manage and choose Nodes > Node Pools in the left-side navigation pane.

3. On the Node Pools page, find the node pool that you want to modify and choose More > Upgrade in the Actions column.

4. Select Change Operating System, select the image that is used to replace the original image, and then click Start Update.

   Note

   By default, Kubelet Update and Upgrade Node Pool by Replacing System Disk are selected when you change the operating system for a node pool. Select Create Snapshot before Update based on your business requirements.

What are the differences between node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter?

The Expected Nodes parameter specifies the number of nodes that you want to keep in a node pool. You can change the value of this parameter to adjust the number of nodes in the node pool. This feature is disabled for existing node pools that are not configured with the Expected Nodes parameter.

Node pools that are configured with the Expected Nodes parameter and those that are not configured with this parameter have different reactions to operations such as removing nodes and releasing ECS instances. The following table shows the details.

How do I add existing nodes to a cluster?

How do I use preemptible instances in a node pool?