Container Service for Kubernetes:Create and manage a Lingjun node pool

Notes before creation

• By default, a Lingjun node pool stores container runtime data on the system disk. Make sure that sufficient free space is reserved for the container runtime directory. Insufficient disk space can cause pods to be evicted or nodes to become abnormal.

• New Lingjun node pools support only the Terway exclusive ENI mode for VPC network communication. Make sure that the limit on the number of pods per node meets your business requirements.

• Nodes in a Lingjun node pool carry the taint node-role.alibabacloud.com/lingjun: NoSchedule by default, which prevents system components (except DaemonSets) from being scheduled to Lingjun nodes and occupying resources. If you want pods to be scheduled to Lingjun nodes, add a toleration for this taint to the pods so that only specific workloads are scheduled to Lingjun nodes. Alternatively, replace it with a custom taint that suits your requirements to achieve the same effect.

  The taint can be removed after the add-ons meet the minimum version requirements. However, after the taint is removed, system components or other workloads may be scheduled to Lingjun nodes and occupy resources of expensive GPU nodes.

• For the complete list of labels that a node pool carries by default, see the description of the node labels setting. Do not remove the default labels.

Procedure

1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

2. Click ... > Create Lingjun Node Pool and follow the on-screen instructions to create the Lingjun node pool.

   After the node pool is created, some settings can be modified by editing the node pool. In the following tables, Modifiable indicates whether a setting can be modified after the node pool is created.

   • Basic settings

     Setting	Description	Modifiable
     Node Pool Name	The custom name of the node pool.	Yes
     Region	By default, the region where the current cluster resides is selected and cannot be changed.	No
     Configure Managed Node Pool	Allowlist feature Enable the managed node pool to use the automated O&M capabilities provided by ACK.	Yes
     Container Runtime	Only containerd is supported.	No

   • Lingjun Resources

     Setting	Description	Modifiable
     Lingjun Cluster	Select a Lite-type Lingjun cluster that is in the same VPC as the ACK cluster.	No
     Lingjun Group	Select a Lingjun group that is in the same VPC as the Lingjun node pool and is not associated with a Lingjun node pool.	No
     Worker RAM Role	Specify a worker RAM role for the Lingjun node pool. After configuration, newly scaled-out nodes in the Lingjun group automatically inherit this role. You must select a Lingjun group first. Only Lingjun instance types that use a cloud disk as the system disk support configuring a worker RAM role. If the selected Lingjun group does not support this feature, this setting is not displayed. Default Role: uses the worker RAM role that is automatically created by ACK and whose trusted entity is Lingjun Intelligent Computing. If this role does not exist in the cluster, it is automatically created and bound to this node pool when you create the node pool. Custom Role: uses a specified RAM role. Only RAM roles whose trust policy includes Lingjun Intelligent Computing (eflo-controller.eflo.aliyuncs.com) as a trusted service can be selected. For more information, see Use custom worker RAM roles. If the selected Lingjun group already has a RAM role configured, this setting automatically selects Custom Role and is fixed to the current RAM role of the group. It cannot be changed or switched back to Default Role. Important If the Lingjun group already has a RAM role configured, the RAM role of the node pool must be consistent with that of the associated Lingjun group. Otherwise, the creation of the Lingjun node pool fails. Newly scaled-out nodes in the group automatically inherit the RAM role of the group. Existing nodes in the group are not affected. After the node pool is created, do not modify the RAM role of the Lingjun group. Otherwise, the node pool may malfunction.	No

   • Volumes

     The custom data disk configuration is an allowlist feature. This feature mounts the container runtime directory to a local data disk. To enable this feature, contact the Container Service team through your customer service manager (PDSA).

     Important

     • During a node pool upgrade, if the storage configurations of nodes are inconsistent, each node is initialized separately based on its actual configuration.

     • To ensure stability, make sure that sufficient free space is reserved for the container runtime directory. Local storage has a risk of data loss. Mount the container runtime directory to a cloud system disk.

   • Advanced Options (Optional)

     View details

   Setting	Description	Modifiable
   System Disk	The cloud system disk configuration of the nodes in the current Lingjun group. When a local system disk is used, the specific configuration is not displayed here. If you do not select Configure local disk mounting for the container runtime directory, the container runtime directory is mounted to the system disk (local system disk or cloud system disk) by default. Important When container runtime data is stored on the system disk, insufficient disk space can cause pods to be evicted or nodes to become abnormal. To ensure stability, make sure that sufficient free space is reserved on the system disk, or configure a dedicated data disk for the container runtime. See Insufficient disk space - DiskPressure for the solutions to insufficient node disk space.	Not applicable
   Data Disk	The local data disks used by Lingjun nodes. After you select Configure local disk mounting for the container runtime directory, a random local data disk is mounted to the container runtime directories, such as /var/lib/container, /var/lib/containerd, /var/lib/kubelet, and /var/log/pods. Configure the Initialization Settings: select the file system format (ext4 or xfs). This takes effect only on unformatted local disks. Important The mount directory must be empty. After a non-empty directory is mounted, the original files become inaccessible. Lingjun node pools support mounting the container runtime directory only to a local data disk, not to a cloud data disk. If the container runtime directory is on a local disk, note the following behavior when a Lingjun node is taken offline for maintenance (standby replacement). The local disk data is not retained, which may cause the container runtime information such as kubelet and containerd to be erased and the node state to become abnormal. To avoid the manual recovery this requires, mount the container runtime directory to a cloud system disk instead. When you enable this feature, do not simultaneously configure the container runtime directory through the ACK Pre-defined Custom Data feature or the Lingjun custom data (UserData) feature. Otherwise, the node may not work as expected.	Yes

   Setting	Description	Modifiable
   Taints	Add key-value taints to the node. A valid taint key consists of an optional prefix and a name. If a prefix is present, it is separated from the name by a forward slash (/). Expand to see detailed instructions Key: The name must be 1 to 63 characters long, start and end with an alphanumeric character [a-z0-9A-Z], and can contain letters, numbers, hyphens (-), underscores (_), and periods (.). If you specify a prefix, it must be a DNS subdomain, which is a series of DNS labels separated by periods (.), not exceeding 253 characters, and ending with a forward slash (/). Value: The taint value can be empty, up to 63 characters long, must start and end with an alphanumeric character [a-z0-9A-Z], and can contain letters, numbers, hyphens (-), underscores (_), and periods (.). Effect: NoSchedule: No new pods that do not tolerate this taint will be scheduled to the node, but running pods are not affected. NoExecute: Not only will no new pods that do not tolerate this taint be scheduled to the node, but any running pods on the node that do not tolerate this taint will be evicted. PreferNoSchedule: ACK will try to avoid scheduling pods to nodes with taints they do not tolerate, but it is not enforced.	Yes
   Node Labels	Important When you create a Lingjun node pool, ACK automatically adds labels to the node pool to enable the exclusive ENI mode. Do not remove them after creation. Do not remove the following default labels of a Lingjun node pool: service.alibabacloud.com/exclude-node alibabacloud.com/lingjun-worker alibabacloud.com/lingjun-hpnzone alibabacloud.com/lingjun-zoneid alibabacloud.com/lingjun-network-type: vpd alibabacloud.com/lingjun-machine-type k8s.aliyun.com/exclusive-mode-eni-type: eniOnly Add key-value labels to the node. A valid Key consists of an optional prefix and a name. If a prefix is present, the prefix and name are separated by a forward slash (/). Expand to see detailed instructions Key: The name must be 1 to 63 characters long, start and end with an alphanumeric character [a-z0-9A-Z], and can contain letters, numbers, hyphens (-), underscores (_), and periods (.). If you specify a prefix, it must be a DNS subdomain, which is a series of DNS labels separated by periods (.), not exceeding 253 characters, and ending with a forward slash (/). The following prefixes are reserved by Kubernetes core components and cannot be specified kubernetes.io/ k8s.io/ prefixes ending with kubernetes.io/ and k8s.io/. For example, test.kubernetes.io/. The following are exceptions: kubelet.kubernetes.io/ node.kubernetes.io prefixes ending with kubelet.kubernetes.io/. prefixes ending with node.kubernetes.io. Value: Can be empty, up to 63 characters long, must start and end with an alphanumeric character [a-z0-9A-Z], and can contain letters, numbers, hyphens (-), underscores (_), and periods (.).	Yes
   Container Image Acceleration	This feature is only supported for containerd runtimes of version 1.6.34 and later. Newly added nodes will automatically detect if a container image supports on-demand loading. If it does, it will use on-demand loading by default to accelerate container startup, reducing application startup time. For more information, see Use on-demand loading of container images to accelerate container startup.	Yes
   Custom Node Name	Whitelisted feature, not applicable to non-Lingjun node pools Description: Directly syncs the hostname of a Lingjun node to its NodeName. Example: If the instance hostname is test, the node's NodeName will be test.	No
   Pre-defined Custom Data	Before a node joins the cluster, the specified pre-customization User-Data script will be run. For example, if you specify the pre-customization data as touch /tmp/pre-script, the combined script on the node will be executed in the following order. #!/bin/bash # The input instance pre-customization data is executed here touch /tmp/pre-script # The ACK node initialization script is executed here	Yes
   User Data	After a node joins the cluster, the specified instance User-Data script will be run. For example, if you specify the instance user data as touch /tmp/post-script, the combined script on the node will be executed in the following order. #!/bin/bash # The ACK node initialization script is executed here # The input instance user data is executed here touch /tmp/post-script Successful cluster creation or node scale-out does not guarantee that the instance user data script executed successfully. You can log on to the node and run grep cloud-init /var/log/messages to view the execution logs.	Yes

Applicable scope

• The node instances to be added must already be in the Lingjun group associated with this node pool and must not have joined other node pools.

• The OperatingState of the Lingjun nodes must be Using.

Procedure

1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

2. On the Node Pools page, click ⋮ Add Existing Nodes and follow the on-screen instructions to complete the following settings:

   • Add Existing Lingjun Node: Select the instances to be added from the instance list. Select Show Unavailable Instances to view unavailable instances.

   • Data Disk: Select how to mount the data disks.

     Available only when the custom data disk configuration allowlist is enabled

     • Configure by Node Pool: Use the data disk configuration preset by the node pool to initialize and mount the data disks.

       If the node pool has no preset data disk configuration, the original configuration of the nodes is retained.

     • No Initialization: Do not initialize the data disks and retain the original data disk configuration of the nodes.

       Important

       • If the storage configurations of nodes are different, during a node pool upgrade, the system initializes the data disks separately based on the actual storage configuration of each node, rather than using a unified configuration.

       • When you use the configure-by-node-pool option, the mount directory must be empty. After a non-empty directory is mounted, the original files become inaccessible.

   After the nodes are added, ACK automatically adds the following labels to the Lingjun node instances. View them in the Lingjun Intelligent Computing console:

   • ack.aliyun.com: the ID of the ACK managed Pro cluster that manages the node.

   • ack.alibabacloud.com/nodepool-id: the ID of the Lingjun node pool that manages the node.

Notes

• Drain nodes:

  • If pods exist on a node, select Drain Node and make sure that the cluster has sufficient resources. The pods are automatically migrated to other nodes.

  • If your applications have Pod Disruption Budgets (PDBs) configured, the draining may fail, which causes the removal to fail. Handle this manually in advance.

  • After draining, if your applications cannot start properly on other nodes, the removal task proceeds regardless.

  • When you remove nodes in an abnormal state, do not select Drain Node. Otherwise, the removal may fail.

• Node state after removal:

  Removing a Lingjun node only removes the node from the Lingjun node pool. It does not remove the node from the Lingjun group, nor does it automatically release or unsubscribe the node.

  Lingjun node instances added to a node pool are not released when the ACK cluster or the Lingjun node pool is deleted, nor are they automatically scaled in from the Lingjun group. Pay attention to the billing status of the Lingjun nodes to avoid additional costs.

  To perform more management operations on nodes and groups, go to the Lingjun Intelligent Computing console.

  The removal is an asynchronous operation. Check the removal progress in the node list.

Procedure

1. On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.

2. Click the name of the target Lingjun node pool, and remove nodes on the Nodes page.

   • Remove a single node: Find the target node, and in the Actions column of the node, choose More > Remove.

   • Batch remove nodes: Select the target nodes and click Batch Remove at the bottom of the page.

3. Choose whether to Drain Node, read the notes on the page, and then follow the on-screen instructions.

Use the RDMA feature

To enable Remote Direct Memory Access (RDMA) network communication for pods on Lingjun nodes, first manually install the ack-rdma-device-plugin add-on on the Add-ons page. For usage instructions, see Use RDMA networks for pods on Lingjun nodes.

Use PPU nodes in a Lingjun node pool

To Use PPUs in a Lingjun node pool, first manually install the ack-rdma-device-plugin and ack-ppu-device-plugin add-ons on the Add-ons page. To use PPU monitoring, install the ppu-device-plugin add-on.

Use PPU supernodes with network topology-aware scheduling

To enable ACK topology-aware scheduling to schedule tasks to PPU supernodes that support ICN Link, first manually install the ack-ppu-device-plugin add-on on the Add-ons page. For usage instructions, see Use Zhenwu PPU M890 supernodes with network topology-aware scheduling.

Why does a node remain in the Not Ready state after maintenance?

• Symptom: A Lingjun node is taken offline for maintenance due to hardware issues, but after the maintenance is complete, the node state in the ACK cluster is still Not Ready.

• Cause: During the offline maintenance of the Lingjun node, machine replacement is performed. The local disk data is not retained, which may cause the container runtime information such as kubelet and containerd to be erased and the node state to become abnormal.

• Solution: After the maintenance is complete, you must manually remove the node from the node pool and re-add it to the Lingjun node pool by adding existing nodes.