This document answers frequently asked questions about deploying VNodes in a self-managed Kubernetes cluster to use Elastic Container Instance (ECI).
-
Networking
-
Image pulling
-
Pod scheduling
-
Storage
-
Logging and monitoring
Accessing on-premises pod IPs from the cloud
If you use Express Connect (a dedicated line) to connect your on-premises and cloud networks, your on-premises data center (IDC) and cloud services can exchange routes using Border Gateway Protocol (BGP). Your on-premises IDC can advertise pod IP routes to cloud services through BGP, allowing cloud services to access the pod IPs in your on-premises IDC. For more information, see Configure BGP.
On-premises access to cloud pod IPs
If you use Express Connect (a dedicated line) to connect your on-premises and cloud networks and have configured BGP, your on-premises data center and cloud services can exchange routes. You can deploy the Cloud Controller Manager (CCM) to automatically synchronize cloud pod IP routes. For more information about CCM, see Cloud Controller Manager.
After you deploy CCM in your self-managed Kubernetes cluster, you can synchronize the Kubernetes pod IP routes to the VPC route table. When you deploy CCM, note the following:
-
You must change the providerID of each Kubernetes cluster node to the
<region-id>.<ecs-id>format, such ascn-shanghai.i-ankb8zjh2nzchf*******. -
Ensure that the IP address of each pod is within the
podCIDRrange of its node. For example, Calico IPAM must be configured ashost-localto ensure the pod IP matches the node's pod CIDR.You can view the pod CIDR range in the node's spec:
spec: podCIDR: 172.23.XX.0/26 podCIDRs: - 172.23.XX.0/26 providerID: cn-shanghai.i-ankb8zjh2nzchfxxxxxxx
Internal domain names cannot be resolved
Problem
Cloud and on-premises services cannot communicate with each other because they are unable to resolve internal domain names. Specific issues include:
-
Cloud services cannot resolve internal domain names in the on-premises data center.
-
Services in the on-premises data center cannot resolve Private DNS (PrivateZone) domain names on the cloud.
Solution
Your on-premises data center (IDC) and Alibaba Cloud VPC are two separate network environments. If services deployed in both locations need to communicate with each other through DNS resolution, you can configure Private DNS (PrivateZone) to enable mutual access by using internal domain names. For more information, see Use PrivateZone and a VPN gateway to access on-premises resources from the cloud.
On-premises access to cloud services
Problem
Services in an on-premises data center (IDC) cannot access Alibaba Cloud services, such as ApsaraDB RDS, Object Storage Service (OSS), or Log Service (SLS), over a dedicated line.
Solution
You can use one of the following methods to resolve this issue. We recommend Method 1.
-
Method 1
Configure cloud service domain names on the cloud. The Virtual Border Router (VBR) will then advertise the routes to your on-premises IDC through BGP. For more information, see Access Cloud Services.
-
Method 2
Configure your IDC to add a static route that directs traffic for the 100.64.0.0/10 CIDR block to the dedicated line.
Image pull failure from self-managed repository
Problem
Pulling an image from a self-managed container image repository fails with the following error:
Failed to pull image "harbor.xxx.com/default/nginx:latest": rpc error: code = Unknown
desc = failed to pull and unpack image "harbor.xxx.com/default/nginx:latest": failed
to resolve reference "harbor.xxx.com/default/nginx:latest": failed to do request: Head
"https://harbor.xxx.com/default/nginx/manifests/latest": x509: certificate is valid
for ingress.local, not harbor-beta.cm.service.163.org
Solution
This error can occur if the image repository uses a self-signed certificate, which causes authentication failure. To skip certificate verification, add the following annotation to the Pod:
"k8s.aliyun.com/insecure-registry": "<host-name>"
For example, if the URL of an NGINX image in your private image repository is test.example.com/test/nginx:apline, you can add the "k8s.aliyun.com/insecure-registry": "test.example.com" annotation to skip certificate verification.
How do I schedule Pods to a VNode?
You can schedule Pods to a VNode, which runs them on ECI, using one of the following common methods:
-
Manual scheduling
You can manually schedule pods to a VNode by configuring nodeSelector and tolerations, or by specifying a nodeName. For more information, see Schedule pods to a VNode.
-
Automatic scheduling
After you deploy the eci-profile component, you can configure a custom selector to automatically schedule matching pods to the VNode. For more information, see Use eci-profile to schedule pods to a VNode.
DaemonSet Pods pending on a VNode
VNodes are virtual nodes and do not support DaemonSets. Therefore, when you create a DaemonSet, you must configure a node anti-affinity policy to prevent its Pods from being scheduled to VNodes. The following is a sample configuration:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
Pod label scheduling failure on VNode
When you schedule a Pod to a VNode using a pod label, ensure that your Kubernetes version is 1.16 or later.
// MutatingAdmissionWebhook does not support ObjectSelector if the Kubernetes version is less than 1.16.
NAS directory mount times out
Cause
When mounting an Apsara File Storage NAS volume, Kubernetes recursively runs chmod and chown on all files in the directory to apply the permissions and ownership from the Pod's security context. If the NAS directory contains a large number of files, this process can cause the volume mount to time out.
Solution
When you configure the security context, set fsGroupChangePolicy to OnRootMismatch. This setting skips the chmod and chown operations if the permissions and ownership of the NAS root directory already match the Pod's specifications. For more information, see Configure a Security Context for a Pod or Container.
Collecting logs for Log Service (SLS)
You can install the Logtail component on your self-managed Kubernetes cluster to collect logs and send them to SLS. The installation process automatically performs the following actions:
-
Creates the
alibaba-log-configurationConfigMap, which contains configuration information for Log Service, such as the Project name. -
(Optional) Creates the AliyunLogConfig Custom Resource Definition (CRD).
-
(Optional) Deploys the
alibaba-log-controllerDeployment, which watches for changes to the AliyunLogConfig CRD and creates Logtail collection configurations. -
Deploys the
logtail-dsDaemonSet, which collects logs from the nodes.
For more information, see Install the Logtail component.
If your cluster runs an earlier version of Kubernetes (for example, 1.13), download and deploy an earlier version of the log collection CRD (alibaba-cloud-log-0.1.1). If you encounter other issues, please submit a ticket.
metrics-server returns a 404 error
VNode currently supports metrics-server 0.5.x and earlier. If you encounter a 404 error, try downgrading the metrics-server version.
Starting from version 0.6.x, metrics-server fetches node metrics from /metrics/resource instead of /stats/summary. VNode does not currently support the /metrics/resource API.
Typically, the startup arguments for metrics-server are as follows:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --kubelet-insecure-tls