You can add existing Elastic Compute Service (ECS) instances to a Container Service for Kubernetes (ACK) cluster or re-add existing nodes to a node pool. To do this, you can log on to the ACK console and manually add nodes or configure ACK to automatically add nodes. Before you start, make sure that you have read the limits and precautions.
Prerequisites
- An ACK cluster is created. For more information, see Create an ACK managed cluster.
A node pool is created in the ACK cluster. For more information, see Procedure.
Limits
Limits on instance types
Limit | Description | Suggested solution |
GPU-accelerated ECS Bare Metal instance | Instances that belong to the GPU-accelerated ECS Bare Metal instance families ebmgn7 and ebmgn7e cannot automatically delete the Multi-Instance GPU (MIG) configuration. ACK automatically resets the MIG configuration retained on instances of the preceding instance families. The reset may be time-consuming. During this period of time, you may fail to add the instances to a cluster. | For more information about how to troubleshoot the issue, see What do I do if I fail to add ECS Bare Metal instances that are equipped with NVIDIA A100 GPUs (ecs.ebmgn7)? |
Quotas
Quota | Description | Suggested solution |
Node quota | Make sure that you have a sufficient node quota in the cluster. For more information about the quotas related to ACK clusters, see Limits. | To increase the node quota, apply for a quota increase in the Quota Center console. |
Limits on ECS instances
Limit | Description | Suggested solution |
Account, region, and VPC | The ECS instances that you want to add must belong to the same Alibaba Cloud account as the cluster. In addition, the ECS instances that you want to add must be deployed in the same region and virtual private cloud (VPC) as your cluster. | Recreate ECS instances to meet the requirement or recreate the cluster. |
Cluster | You cannot add ECS instances that already belong to other clusters. | If you want to add ECS instances that already belong to other clusters, remove the ECS instances from the cluster and then add them to the desired cluster. For more information, see Remove a node. |
Limits on networks
Limit | Description | Suggested solution |
Terway | Terway has the following limits on the maximum number of pods supported by the type of ECS instances to be added.
The maximum number of pods supported in each ENI mode depends on the maximum number of ENIs supported by the instance type. For more information about how to calculate the maximum number of pods supported in each ENI mode, see Compare Terway modes. For more information about how to query the maximum number of pods supported by an instance type, see View the maximum number of pods supported by the node network. |
|
Terway has the following limits on ENIs. | ||
When you add a node in a new zone to the cluster, you must update the vSwitch configuration of Terway. Otherwise, IP addresses are allocated to the pods on the node from the vSwitch of the primary ENI of the node. | Increase the number of pod vSwitches in a cluster that uses the Terway plug-in. | |
When you add an existing node to a node pool, the ENI that is bound to the node is retained. The vSwitch to which the ENI belongs is used to assign IP addresses to the pods on the node. Make sure that the node to be added has only one primary ENI. | If pods on a node use IP addresses that do not belong to the vSwitch of the node, remove the node from the cluster, delete all secondary ENIs, and then re-add the node to the cluster. | |
When you add an existing node to a node pool, make sure that the cluster RAM role is assigned to the corresponding ECS instance. Otherwise, the system cannot correctly calculate the maximum number of pods (MaxPod) supported by the node due to permissions issues. Consequently, the maximum number of ENIs calculated by the system is incorrect. | ||
Flannel | The number of custom route entries in the system route table of the VPC where a cluster resides cannot exceed the quota limit of the route table. For more information about custom route entries, see Route table overview. | Apply for a quota increase in the Quota Center console. |
IPv4/IPv6 dual-stack | You must assign an IPv6 address to the primary ENI of the ECS instance to be added. | For more information, see Configure an IPv6 address for an ECS instance. |
Limits on security groups
Limit | Description | Suggested solution |
Security group type | When you add an existing ECS instance to a node pool, the instance is also added to the security group of the node pool. Due to ECS limits, an ECS instance cannot belong to a basic security group and an advanced security group at the same time. Therefore, the type of security group to which an ECS instance belongs must be the same as that of the node pool to which the ECS instance is added. You can view the security group of a node pool on the Overview tab of the node pool details page in the ACK console. You can also go to the ECS console to view the security group to which an ECS instance belongs. For more information about security groups, see Overview. | You cannot modify the security groups of node pools or change the type of security group. If a security group conflict occurs, use the following solutions.
|
Security group rules | The security group rules of an ECS instance must not conflict with the security group rules of the node pool and the cluster to which the ECS instance is added. | Add existing nodes to the security group of the desired node pool first. Use the security group rule check feature to check each security group rule of the node pool. |
Number of security groups | When you add an existing ECS instance to a node pool, the instance is also added to the security group of the node pool. Make sure that the number of security groups to which the instance belongs does not exceed the upper limit. The default upper limit is 5. | For more information about security group limits and how to increase the quota limit of security groups for your ECS instance, see Security group limits. |
Limits on operating systems
Only ECS instances that run Alibaba Cloud Linux, CentOS, and Red Hat are supported. We recommend that you choose Alibaba Cloud Linux 3. For more information about Alibaba Cloud Linux 3 and its benefits, see Use Alibaba Cloud Linux 3.
Precautions
Instance release
When you delete a cluster or node pool, ECS instances that are added to the cluster or node pool are not automatically released. You need to manually release the ECS instances. For more information, see Remove a node.
Billing
ECS instances that you add to a cluster are not released when you delete the cluster or node pools in the cluster. To avoid unnecessary costs, we recommend that you check the billing of the ECS instances in a timely manner. For more information, see Billing overview.
Instance configuration
When you add existing ECS instances to a node pool, the billing method and instance type of the ECS instances remain unchanged.
Operating systems and disks
We recommend that you back up data before you add ECS instances in case data is lost or damaged. For more information, see Create a snapshot for a disk.
If you configure ACK to automatically add a node, ACK replaces the operating system of the node with the operating system of the node pool. If you manually add a node, the operating system of the node is retained. To retain the operating system of the node that you want to add, manually add the node.
If you configure ACK to automatically add a node, the system disk of the node is released but the data disk is retained. However, the ID of the data disk is changed. Data in the data disk is not affected.
If you configure ACK to automatically add a node, the user snapshots of the system disk of the node are retained. Automatic snapshots are retained or deleted together with the system disk based on the setting of the Delete Automatic Snapshots While Releasing Disk attribute of the system disk. You can go to the disk details page in the ECS console and click Modify Attributes to view or modify the setting.
To ensure that you have a sufficient snapshot quota to periodically run automatic snapshot policies, we recommend that you delete user snapshots and automatic snapshots that are no longer needed.
Automatically or manually add nodes
Automatically or manually adding nodes does not affect the existing nodes or applications in your cluster. If you configure ACK to automatically add a node, ACK replaces the operating system of the node with the operating system of the node pool. If you manually add a node, the operating system of the node is retained. To retain the operating system of an ECS instance, manually add the ECS instance.
To avoid compatibility issues, we recommend that you do not initialize ECS instances as worker nodes if services are already deployed on the ECS instances.
Automatically add nodes
When you configure ACK to automatically add nodes, all available ECS instances within the current account are displayed in the ACK console. After you configure parameters, the ECS instances are automatically added to your cluster.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Node Pools page, find the node pool that you want to manage, click More in the Actions column, and then click Add Existing Node.
On the Select Existing ECS Instance wizard page, set Mode to Auto, select existing ECS instances in the list, and click Next Step.
ImportantIf no ECS instance is displayed in the list, this means that the existing ECS instances do not meet the requirement. Refer to Limits and Precautions and locate the cause.
You can select Show Unavailable Instances to view the reason why the ECS instances cannot be added to the cluster. If no ECS instance is displayed after you select the check box, check whether the existing ECS instances reside in the same region and VPC as the cluster.
On the Specify Instance Information wizard page, configure parameters and click Next Step.
Parameter
Description
Cluster ID/Name
Information about the cluster to which you want to add the ECS instances. This parameter is automatically set.
Data Disk
Specify whether to store the container and image data on a data disk.
If the ECS instances have data disks mounted and the file system of the last data disk is not initialized, the system automatically formats the data disk to ext4. Then, the system uses the disk to store the data in /var/lib/docker (the default data directory of Docker) and /var/lib/kubelet (the default data directory of the kubelet).
ImportantData on a disk is lost after you format the disk. Before the system formats the data disk, we recommend that you back up the data on the data disk.
If no data disk is attached to the ECS instances, the system does not purchase a data disk.
Retain Instance Name
By default, Retain Instance Name is turned on. If you do not want to retain the instance name, you can turn off Retain Instance Name. After you disable this feature, the nodes are renamed based on the node naming rules.
Instance Information
The IDs and names of the ECS instances that you want to add.
In the Confirm message, read the precautions and click Confirm.
After the ECS instances are added, you can click Details in the Actions column of the node pool on the Node Pools page to view the information about the newly added nodes on the Nodes tab.
Manually add nodes
Manually adding nodes does not change the operating systems of the nodes. In addition to Limits, you also need to pay attention to the following limits before you manually add nodes.
Make sure that swap is disabled for the operating systems of the ECS instances that you want to add.
If you want to store container data and images on a data disk, make sure that the data disk uses the ext or XFS file system.
In manual mode, you must obtain the installation command, log on to an ECS instance, and then run the command to add the ECS instance to an ACK cluster. You can add only one ECS instance at a time.
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Node Pools page, find the node pool that you want to manage, click More in the Actions column, and then click Add Existing Node.
On the Select Existing ECS Instance wizard page, set Mode to Manual, select existing ECS instances in the list, and click Next Step.
On the Specify Instance Information wizard page, configure parameters and click Next Step.
Parameter
Description
Cluster ID/Name
Information about the cluster to which you want to add the ECS instances. This parameter is automatically set.
Data Disk
Specify whether to store the container and image data on a data disk.
If the ECS instances have data disks mounted and the file system of the last data disk is not initialized, the system automatically formats the data disk to ext4. Then, the system uses the disk to store the data in /var/lib/docker (the default data directory of Docker) and /var/lib/kubelet (the default data directory of the kubelet).
ImportantData on a disk is lost after you format the disk. Before the system formats the data disk, we recommend that you back up the data on the data disk.
If no data disk is attached to the ECS instances, the system does not purchase a data disk.
Retain Instance Name
By default, Retain Instance Name is turned on. If you do not want to retain the instance name, you can turn off Retain Instance Name. After you disable this feature, the nodes are renamed based on the node naming rules.
Instance Information
The IDs and names of the ECS instances that you want to add.
On the Complete wizard page, copy the command that is used to remotely log on to the ECS instance and click Done.
Log on to the ECS console. In the left-side navigation pane, choose Instances & Images > Instances. Then, select the region where your cluster is deployed and select the ECS instance that you want to add.
Click Connect in the Actions column. In the Connection and Command dialog box, select a connection method and go to the connection page.
The following table describes the connection methods.
Connection method
Description
VNC Connection
For more information about how to use VNC to remotely connect to an ECS instance, see Connect to an instance by using VNC.
Send Remote Commands (Cloud Assistant)
This is the recommended method. This method allows you to run remote commands on an instance. You can perform operations such as viewing disk capacity, installing software, and starting or stopping services without logging on to the instance. This feature is implemented by using Cloud Assistant. For more information about how to install or activate Cloud Assistant, see Install Cloud Assistant Agent.
On the connection page, follow the instructions and paste the command that you copied in Step 6. Then, click Run to execute the script.
After the script is executed, the ECS instance is added to the cluster.
After the ECS instance is added, you can click Details in the Actions column of the node pool on the Node Pools page to view the information about the newly added node on the Nodes tab.
FAQ
Are businesses in a cluster affected if I upgrade or downgrade the ECS instances that I add to the cluster?
ECS instance upgrades or downgrades usually involve changes in the ECS instance type, Internet bandwidth, billing method of the Internet bandwidth, and billing method of the data disk. For more information, see Overview of instance configuration changes. Different upgrade or downgrade methods have different impacts on ECS instances.
For upgrades or downgrades that do not require a restart, you need to assess the impacts based on the actual business scenario.
Upgrades or downgrades that require a restart, such as instance type changes, will adversely affect your businesses. Before you perform the operation (upgrade worker node resources), you need to confirm whether you need to add additional nodes to host the evicted application pods. Then, drain the node that you want to upgrade or downgrade, and remove it from the scaling group and ACK cluster. For more information, see Remove a node.
After the upgrade or downgrade is complete, perform the steps in this topic to re-add the node to the cluster.
What do I do if a timeout error occurs after I add an existing node?
Check whether the network of the node and the network of the Classic Load Balancer (CLB) instance of the API server are connected. Check whether the security groups meet the requirement. For more information about the limits on security groups, see Limits on security groups. For more information about other network connectivity issues, see FAQ about network management.
Can I add existing nodes of different instance types to the same ACK cluster?
Yes, you can add nodes of different instance types to a node pool. This avoids node scale-out failures when the specified instance type is unavailable or ECS instances of the specified type are out of stock. To do this, perform the following steps:
Create or modify a node pool and select the desired instance types. For more information, see Create a node pool and Modify a node pool.
Refer to Remove a node to remove and drain the nodes. Do not release the corresponding ECS instances.
Refer to Limits and Automatically or manually add nodes and add ECS instances of different instance types to the node pool.
How do I move nodes across ACK clusters?
You cannot directly move nodes across ACK clusters. You need to use the method for adding existing nodes. To do this, perform the following steps:
Refer to Remove a node to remove and drain the nodes. Do not release the corresponding ECS instances.
Refer to Limits and Automatically or manually add nodes and add ECS instances of different instance types to the node pool.
References
In addition to the console, you can also use the API or CLI to add ECS instances to an ACK cluster.
Free nodes may exist in ACK clusters that are created before the node pool feature is released. Free nodes refer to cluster nodes that do not belong to any node pool. You can refer to Add free nodes to a node pool to configure ACK to manage these nodes.
For more information about the parameters of managed node pools, see Overview of managed node pools and Create a node pool.
For more information about how to troubleshoot node or pod exceptions, see Node troubleshooting, Pod troubleshooting, and FAQ about nodes and node pools.