Common issues for adding edge nodes - Container Service for Kubernetes

Distinguishing between cloud and edge nodes

ACK Edge uses the alibabacloud.com/is-edge-worker label to identify edge nodes.

When you add a node to a cloud node pool or an edge node pool, the is-edge-worker label is automatically added. If the is-edge-worker value is true, the node is an edge node. If the value is false, the node is a cloud node.

Add edge nodes over Express Connect

To add an edge node to an ACK Edge cluster over Express Connect, meet these requirements. See Special configurations for ACK Edge clusters in Express Connect scenarios.

When you create an edge node pool, select Dedicated for the node pool type. Then, follow Add an edge node to generate a node connection script.

See Manage edge node pools.

Note
If your ACK Edge cluster is version 1.22 or later, you cannot use the inDedicatedNetwork parameter in the node connection script to add nodes over Express Connect. If your cluster is earlier than 1.22, upgrade it.
In a dedicated line scenario, node onboarding interacts with cloud services over internal endpoints. Ensure network connectivity to required services such as OSS, Container Registry, and Server Load Balancer (SLB).

Add GPU-accelerated nodes

Install the GPU driver before you add the node.
See NVIDIA driver versions supported by ACK.

Set the gpuVersion parameter in the node connection script to match your GPU model.

OS architecture	GPU model	ACK Edge cluster version
AMD64/x86_64	Nvidia_Tesla_T4	≥1.16.9-aliyunedge.1
AMD64/x86_64	Nvidia_Tesla_P4	≥1.16.9-aliyunedge.1
AMD64/x86_64	Nvidia_Tesla_P100	≥1.16.9-aliyunedge.1
AMD64/x86_64	Nvidia_Tesla_V100	≥1.18.8-aliyunedge.1
AMD64/x86_64	Nvidia_Tesla_A10	≥1.20.11-aliyunedge.1
AMD64/x86_64	Nvidia_L40	≥1.26.3-aliyun.1

In the node configuration file, set the gpuVersion field to your GPU model, such as Nvidia_Tesla_T4:

{
  "gpuVersion": "Nvidia_Tesla_T4",
  "enableIptables": true,
  "quiet": true,
  "manageRuntime": true,
  "allowedClusterAddons": [
    "kube-proxy",
    "flannel",
    "coredns"
  ]
}

After configuration, the connection tool automatically installs nvidia-containerd-runtime. See NVIDIA Container Runtime.

Troubleshoot node connection script failures

If the script fails, use the following table to troubleshoot. If the issue is not listed, collect the node diagnostic information and submit a ticket. See How to collect diagnostic information for a node in an ACK Edge cluster.

Error message	Cause	Solution
The os XXX unsupport	The edge node OS is not supported.	For supported operating systems, see Add an edge node.
invalid nodeName	The node name is invalid.	The node name can contain only lowercase letters, hyphens (-), and periods (.). The node name cannot exceed 253 characters. The node name cannot start with "localhost".
Node route overlaps with service cidr	The node route table conflicts with the pod or service CIDR block configured during cluster creation.	Recreate the cluster. Ensure the pod and service CIDR blocks do not conflict with the NameServer address or edge node route table.
response error msg: TOKEN_EXPIRED	The connection token has expired.	Regenerate the node connection script. Check whether the system time on the node is correct.
A node named XXX already exists in the cluster	A node with the same name already exists in the cluster.	Remove the node with the same name from the cluster.
error run phase join-node: failed to get cluster info: failed to get cluster-info configmap, Get "https://xx.xxx.xx.xx:6443/api/v1/namespaces/kube-public/configmaps/cluster-info": dial tcp xx.xxx.xx.xx:6443: i/o timeout	Failed to get the cluster-info ConfigMap.	To add edge nodes, edgeadm accesses the API server at this address. Check whether the access control list (ACL) rules of the API server SLB instance restrict this access.
error run phase join-node: Install edge-hub failed: Copy file /tmp/edge-hub to /usr/bin/edge-hub fail: open /usr/bin/edge-hub: text file busy \| 40009 \| 40009	The edge-hub binary already exists on the node.	Run `edgeadm reset` to clean up the node, then re-add it.
error run phase post-check: timed out waiting for the condition	A system component failed to start.	Download the latest edgeadm. Run `edgeadm reset` and re-add the node. Verify the edge node can access its required public endpoints. See Network management. Collect the edge node diagnostic information and submit a ticket. See How to collect diagnostic information for a node in an ACK Edge cluster.

Edge node upgrade failures

When you upgrade an edge node pool, if the This node has been upgraded successfully message is not returned, use the following table to troubleshoot.

Error message	Cause	Solution
edgeadm version xxxx does not match cluster version	The upgrade tool version does not match the cluster version.	Verify the cluster control plane is upgraded. Verify `TARGET_CLUSTER_VERSION` is configured correctly.
node has already been upgraded to xxx	The node is already at the target version.	If some node components were not upgraded, save the logs and submit a ticket.
kubelet target version xxxx does not match cluster version xxxx	The specified kubelet version for the upgrade does not match the cluster control plane version.	If you specified `kubelet-version`, verify its value matches the cluster control plane version. If you did not specify this parameter, submit a ticket for assistance.
Parameter currentVersion cann't null	An outdated edgeadm version was used.	Verify edgeadm is the latest version. Supported cluster upgrades are from version 1.18 to 1.20 and from 1.20 to 1.22.
upgrade kubelet failed at phase install, recover to previous state. error run phase upgrade: xxxx	The upgrade failed and automatically rolled back. Node status is unaffected.	Save the logs and submit a ticket.
upgrade kubelet failed at phase install, recover to previous state recover kubelet failed, err: xxx error run phase upgrade: xxxx	The upgrade and automatic rollback both failed. Node status is affected.	Save the logs and submit a ticket.

Collect ACK Edge cluster diagnostic information for a node

If a node in your ACK Edge cluster has an issue, follow these steps to collect its diagnostic information.

Log on to the affected node in the ACK Edge cluster.

Download the diagnostic script.

curl -o /usr/local/bin/diagnose_edge_node.sh https://aliacs-k8s-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/public/diagnose/diagnose_k8s.sh

Make the diagnostic script executable.

chmod u+x /usr/local/bin/diagnose_edge_node.sh

Change to the script directory:
```
cd /usr/local/bin/
```

Run the diagnostic script.

./diagnose_edge_node.sh

Sample output. Each run generates a diagnostic file with a unique name, such as diagnose_1578310147.tar.gz.

......
+ echo 'please get diagnose_1578310147.tar.gz for diagnostics'
please get diagnose_1578310147.tar.gz for diagnostics
+ echo 'Please submit diagnose_1578310147.tar.gz to technical support'
Please submit diagnose_1578310147.tar.gz to technical support

Run ll to verify the diagnose_1578310147.tar.gz file exists.