To add an ECS instance as a worker node to an ACK cluster or re-add a worker node to the node pool, configure ACK to automatically add the node or manually add the node in the ACK console. When you add the node to the node pool, the configuration of the node pool is automatically used by the node, and you can centrally manage the nodes at the node pool level.
Limits
Limits on instance types
Limit | Description | Suggested solution |
GPU-accelerated ECS Bare Metal instance | Instances that belong to the GPU-accelerated ECS Bare Metal instance families ebmgn7 and ebmgn7e cannot automatically delete the Multi-Instance GPU (MIG) configuration. ACK automatically resets the MIG configuration retained on instances of the preceding instance families. The reset may be time-consuming. During this period of time, you may fail to add the instances to a cluster. | For more information about how to troubleshoot the issue, see What do I do if I fail to add ECS Bare Metal instances that are equipped with NVIDIA A100 GPUs? |
Quotas
Limit | Description | Suggested solution |
Node quota | Make sure that you have a sufficient node quota in the cluster. For more information about the quotas related to ACK clusters, see Limits. | To increase the node quota, apply for a quota increase in the Quota Center console. |
Limits on ECS instances
Limit | Description | Suggested solution |
Account, region, and VPC | The ECS instances that you want to add must belong to the same Alibaba Cloud account as the cluster. In addition, the ECS instances that you want to add must be deployed in the same region and virtual private cloud (VPC) as your cluster. | Recreate ECS instances to meet the requirement or recreate the cluster. |
Cluster | You cannot add ECS instances that already belong to other clusters. | If you want to add ECS instances that already belong to other clusters, remove the ECS instances from the cluster and then add them to the desired cluster. For more information, see Remove a node. |
Limits on networks
Limit | Description | Suggested solution |
Terway | Terway has the following limits on the maximum number of pods supported by the type of ECS instances to be added.
The maximum number of pods supported in each ENI mode depends on the maximum number of ENIs supported by the instance type. For more information about how to calculate the maximum number of pods supported in each ENI mode, see Work with Terway. For more information about how to query the maximum number of pods supported by an instance type, see Work with Terway. |
|
Terway has the following limits on ENIs. | ||
When you add a node in a new zone to the cluster, you must update the vSwitch configuration of Terway. Otherwise, IP addresses are allocated to the pods on the node from the vSwitch of the primary ENI of the node. | ||
When you add an existing node to a node pool, the ENI that is bound to the node is retained. The vSwitch to which the ENI belongs is used to assign IP addresses to the pods on the node. Make sure that the node to be added has only one primary ENI. | If pods on a node use IP addresses that do not belong to the vSwitch of the node, remove the node from the cluster, delete all secondary ENIs, and then re-add the node to the cluster. | |
When you add an existing node to a node pool, make sure that the cluster RAM role is assigned to the corresponding ECS instance. Otherwise, the system cannot correctly calculate the maximum number of pods (MaxPod) supported by the node due to permissions issues. Consequently, the maximum number of ENIs calculated by the system is incorrect. | Use RBAC to manage the operation permissions on resources in a cluster. | |
Flannel | The number of custom route entries in the system route table of the VPC where a cluster resides cannot exceed the quota limit of the route table. For more information about custom route entries, see Route tables. | Apply for a quota increase in the Quota Center console. |
IPv4/IPv6 dual-stack | You must assign an IPv6 address to the primary ENI of the ECS instance to be added. | For more information about how to assign an IPv6 address to the primary ENI of an ECS instance, see IPv6 communication. |
Limits on security groups
Limit | Description | Suggested solution |
Security group type | When you add an existing ECS instance to a node pool, the instance is also added to the security group of the node pool. Due to ECS limits, an ECS instance cannot belong to a basic security group and an advanced security group at the same time. Therefore, the type of security group to which an ECS instance belongs must be the same as that of the node pool to which the ECS instance is added. You can view the security group of a node pool on the Overview tab of the node pool details page in the ACK console. You can also go to the ECS console to view the security group to which an ECS instance belongs. For more information about security groups, see Overview. | You cannot modify the security groups of node pools or change the type of security group. If a security group conflict occurs, use the following solutions.
|
Security group rules | The security group rules of an ECS instance must not conflict with the security group rules of the node pool and the cluster to which the ECS instance is added. | Add existing nodes to the security group of the desired node pool first. Use the security group rule check feature to check each security group rule of the node pool. |
Number of security groups | When you add an existing ECS instance to a node pool, the instance is also added to the security group of the node pool. Make sure that the number of security groups to which the instance belongs does not exceed the upper limit. The default upper limit is 5. | For more information about security group limits and how to increase the quota limit of security groups for your ECS instance, see Security group limits. |
Precautions
Instance release
When you delete a cluster or node pool, ECS instances that are added to the cluster or node pool are not automatically released. You need to manually release the ECS instances. For more information, see Remove a node.
Billing
ECS instances that you add to a cluster are not released when you delete the cluster or node pools in the cluster. To avoid unnecessary costs, we recommend that you check the billing of the ECS instances in a timely manner. For more information, see Billing overview.
Node pools and instance configurations
When you add existing ECS instances to a node pool, the billing method and instance type of the ECS instances remain unchanged.
This feature does not support node pools whose Scaling Mode is set to Auto.
Operating systems and disks
We recommend that you back up data before you add ECS instances in case data is lost or damaged. For more information, see Create a snapshot.
If you configure ACK to automatically add a node, ACK resets the operating system of the node with the operating system of the node pool. If you manually add a node, the operating system of the node is retained. To retain the operating system of the node that you want to add, manually add the node.
If you configure ACK to automatically add a node, the system disk of the node is released but the data disk is retained. However, the ID of the data disk is changed. Data in the data disk is not affected.
If you configure ACK to automatically add a node, user snapshots of the system disk are retained. Automatic snapshots are retained or deleted together with the system disk based on the setting of the Delete Automatic Snapshots While Releasing Disk attribute of the system disk. You can go to the disk details page in the ECS console and click Modify Attributes to view or modify the setting.
To ensure that you have a sufficient snapshot quota to periodically run automatic snapshot policies, we recommend that you delete user snapshots and automatic snapshots that are no longer needed.
Automatically or manually add nodes
Automatically or manually adding nodes does not affect the existing nodes or applications in your cluster. To avoid compatibility issues, we recommend that you do not initialize ECS instances as worker nodes if services are already deployed on the ECS instances.
If you configure ACK to automatically add a node, ACK replaces the operating system of the node with the operating system of the node pool. If you manually add a node, the operating system of the node is retained.
If you manually add a node, the original operating system of the nodes is retained. The operating system is not reset. In addition to Limits, you also need to pay attention to the following limits before you manually add nodes.
You cannot add an ECS instance that has an operating system of ContainerOS. For more information, see ContainerOS.
Make sure that swap is disabled for the operating system of the ECS instance that you want to add.
If you want to store container data and images on a data disk, make sure that the data disk uses the ext or XFS file system.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster to manage and click its name. In the left-side navigation pane, choose .
On the Node Pools page, find the node pool that you want to manage, click More in the Actions column, and then click Add Existing Node.
Manually add nodes
You must obtain the installation command, log on to an ECS instance, and then run the command to add the ECS instance to an ACK cluster. You can add only one ECS instance at a time.
On the Select Existing ECS Instance wizard page, set Mode to Manual, select existing ECS instances in the list, and click Next Step.
On the Specify Instance Information wizard page, configure parameters and click Next Step.
Parameter
Description
Cluster ID/Name
Information about the cluster to which you want to add the ECS instances. This parameter is automatically set.
Data Disk
Specify whether to store the container and image data on a data disk.
If the ECS instances have data disks mounted and the file system of the last data disk is not initialized, the system automatically formats the data disk to ext4. Then, the system uses the disk to store the data in /var/lib/docker (the default data directory of Docker) and /var/lib/kubelet (the default data directory of the kubelet).
ImportantData on a disk is lost after you format the disk. Before the system formats the data disk, we recommend that you back up the data on the data disk.
If no data disk is attached to the ECS instances, the system does not purchase a data disk.
Retain Instance Name
By default, Retain Instance Name is turned on. If you do not want to retain the instance name, you can turn off Retain Instance Name. After you disable this feature, the nodes are renamed based on the node naming rules.
Instance Information
The IDs and names of the ECS instances that you want to add.
On the Complete wizard page, copy the command that is used to remotely log on to the ECS instance and click Done.
Log on to the ECS console. In the left-side navigation pane, choose Instances & Images > Instances. Then, select the region where your cluster is deployed and select the ECS instance that you want to add.
Click Connect in the Actions column. In the Connection and Command dialog box, select a connection method and go to the connection page.
For more information about how to connect to an instance, see Methods for connecting to an ECS instance.
On the connection page, follow the instructions and paste the command that you saved. Then, click Run to execute the script.
After the script is executed, the ECS instance is added to the cluster.
After the ECS instance is added, you can click Details in the Actions column of the node pool on the Node Pools page to view the information about the newly added node on the Nodes tab.
Automatically add nodes
All available ECS instances within the current account are displayed in the ACK console. After you configure parameters, the ECS instances are automatically added to your cluster.
On the Select Existing ECS Instance wizard page, set Mode to Auto, select existing ECS instances in the list, and click Next Step.
ImportantIf no ECS instance is displayed in the list, this means that the existing ECS instances do not meet the requirement. Refer to Limits and Precautions and locate the cause.
You can select Show Unavailable Instances to view the reason why the ECS instances cannot be added to the cluster. If no ECS instance is displayed after you select the check box, check whether the existing ECS instances reside in the same region and VPC as the cluster.
On the Specify Instance Information wizard page, configure parameters and click Next Step.
Parameter
Description
Cluster ID/Name
Information about the cluster to which you want to add the ECS instances. This parameter is automatically set.
Data Disk
Specify whether to store the container and image data on a data disk.
If the ECS instances have data disks mounted and the file system of the last data disk is not initialized, the system automatically formats the data disk to ext4. Then, the system uses the disk to store the data in /var/lib/docker (the default data directory of Docker) and /var/lib/kubelet (the default data directory of the kubelet).
ImportantData on a disk is lost after you format the disk. Before the system formats the data disk, we recommend that you back up the data on the data disk.
If no data disk is attached to the ECS instances, the system does not purchase a data disk.
Retain Instance Name
By default, Retain Instance Name is turned on. If you do not want to retain the instance name, you can turn off Retain Instance Name. After you disable this feature, the nodes are renamed based on the node naming rules.
Instance Information
The IDs and names of the ECS instances that you want to add.
In the Confirm message, read the precautions and click Confirm.
After the ECS instance is added, you can click Details in the Actions column of the node pool on the Node Pools page to view the information about the newly added node on the Nodes tab.
FAQ
Are businesses in a cluster affected if I upgrade or downgrade the ECS instances that I add to the cluster?
ECS instance upgrades or downgrades usually involve changes in the ECS instance type, Internet bandwidth, billing method of the Internet bandwidth, and billing method of the data disk. For more information, see Overview of instance configuration changes. Different upgrade or downgrade methods have different impacts on ECS instances.
For upgrades or downgrades that do not require a restart, you need to assess the impacts based on the actual business scenario.
Upgrades or downgrades that require a restart, such as instance type changes, will adversely affect your businesses. Before you perform the operation (upgrade or downgrade worker node resources), you need to confirm whether you need to add additional nodes to host the evicted application pods. Then, drain the node that you want to upgrade or downgrade, and remove it from the scaling group and ACK cluster. For more information, see Remove a node.
After the upgrade or downgrade is complete, perform the steps in this topic to re-add the node to the cluster.
What do I do if a timeout error occurs after I add an existing node?
Check whether the network of the node and the network of the Classic Load Balancer (CLB) instance of the API server are connected. Check whether the security groups meet the requirement. For more information about the limits on security groups, see Limits on security groups. For more information about other network connectivity issues, see FAQ about network management.
Can I add existing nodes of different instance types to the same ACK cluster?
Yes, you can add nodes of different instance types to a node pool. This avoids node scale-out failures when the specified instance type is unavailable or ECS instances of the specified type are out of stock. To do this, perform the following steps:
Create or modify a node pool and select the desired instance types. For more information, see Create and manage a node pool.
Refer to Remove a node to remove and drain the nodes. Do not release the corresponding ECS instances.
Refer to Limits and Automatically or manually add nodes and add ECS instances of different instance types to the node pool.
How do I move nodes across ACK clusters?
You cannot directly move nodes across ACK clusters. You need to use the method for adding existing nodes. To do this, perform the following steps:
Refer to Remove a node to remove and drain the nodes. Do not release the corresponding ECS instances.
Refer to Limits and Automatically or manually add nodes and add ECS instances of different instance types to the node pool.
Does the expected number of nodes in the node pool automatically change after an existing node is added?
After you add an existing node, the expected number of nodes is automatically increased by the number of added nodes. For example, the value of the Expected Nodes parameter of your node pool is set to 5. If you add an existing ECS instance to the node pool, the value of the Expected Nodes parameter of your node pool automatically changes to 6.
References
In addition to the ACK console, you can also call API operations or use CLI to add ECS instances to an ACK cluster. For more information, see Manually add an existing instance to a specified node pool and Add existing ECS instances.
Free nodes exist in clusters created before the node pool feature was released. Free nodes are nodes that do not belong to any node pool. For more information about how to add free nodes to a node pool, see Add free nodes to a node pool.
For more information about how to troubleshoot node or pod exceptions, see Troubleshoot node exceptions, Pod troubleshooting, and FAQ about nodes and node pools.