Enable internal domain name access to the API Server for an existing cluster - Container Service for Kubernetes

When you enable internal domain name resolution for an existing ACK Pro cluster, node components such as kubelet and kube-proxy can directly access the API Server by dynamically resolving its endpoint through an internal domain name. This method is an optimization over accessing the API Server through the fixed IP address of a Server Load Balancer (SLB). This improves the performance and availability of node component access to the API Server in large-scale clusters.

Scope

This feature is available only for ACK Pro clusters that run Kubernetes version 1.30 or later.
This feature cannot be enabled for hybrid cloud node pools.
The API Server internal domain name is enabled by default for the following clusters. You do not need to enable it manually.
- Clusters that are created after November 2024 and run Kubernetes version 1.20 or later.
- If the internal_dns_config.enabled parameter is set to true when you call the DescribeClusterDetail operation, the feature is enabled.

Changes and impacts

Network path changes and security group impacts:
After you enable this feature, the communication path for nodes to access the API Server switches from the fixed IP address of the API Server SLB to a direct connection through the internal domain name.
- Before the change (SLB forwarding): Node -> SLB IP -> API Server
- After the change (direct connection through internal domain name): Node -> (Internal domain name resolution) -> API Server Elastic Network Interface (ENI)
This means traffic bypasses the SLB and directly accesses the API Server ENI. Therefore, ensure that the security group of the node allows traffic to and from the security group of the ENI.
Public network access dependencies: After you enable this feature, the default outbound route for the API Server changes to the cluster's VPC. If the cluster is configured with an OIDC Issuer URL or an external webhook that depends on a public IP address, ensure that the cluster's VPC has public network access. Otherwise, the API Server cannot connect to external services.
Scope of the configuration:
- After you enable this feature, the path for kubelet and kube-proxy to access the API Server switches from the fixed IP address of the SLB to a direct connection to the API Server ENI.
- The kubernetes service endpoints in the `default` namespace will switch from pointing to the API Server SLB IP address to the API Server ENI IP address. This allows applications such as Operators within the cluster to connect directly.

Prerequisites

Before you begin, perform the following checks. Otherwise, nodes cannot connect to the API Server.

1. Check security group configurations

After you enable a direct connection, traffic goes directly to the API Server's ENI. Ensure that the security group rules for the ENI's security group allow direct connections. For more information, see Manage security groups.

Security group type	Check item	Recommended action
Enterprise security group	Inbound rule	You must explicitly allow traffic. Manually add a rule to allow access to TCP port 6443 from the cluster's VPC CIDR block and all secondary CIDR blocks.
Basic security group	Intra-group connectivity	If the node or container is in the same security group as the API Server, they can communicate by default. If they are in different security groups, configure the security groups to allow traffic between them or allow access to port 6443 from the corresponding CIDR block.

Check the outbound rules of the security groups for the node and the node's ENI to ensure that access to port 6443 is not denied.

2. Check for webhook and OIDC public network dependencies

Check for webhooks that depend on public URLs.

kubectl get validatingwebhookconfiguration -o yaml -A | grep url
kubectl get mutatingwebhookconfiguration -o yaml -A | grep url

On the ACK Clusters page, click the name of the target cluster. In the navigation pane on the left, click Add-ons. Locate the API Server and check the OIDC-related parameters in its configuration to confirm whether a public address is used.

If the API Server has the public network access dependencies mentioned above and the cluster's VPC does not have a public network egress, you must first configure a NAT Gateway to provide a unified egress for Internet traffic.

3. Check DNS resolution configurations

API Server domain name resolution depends on the default internal DNS servers of the VPC (100.100.2.136 and 100.100.2.138).

Default configuration: If nodes use the default DNS configuration, no additional action is required.
Custom DNS: If you configure a custom DNS server on a node or container, ensure that the DNS server can forward requests for *.aliyuncs.com or the API Server domain name to the Alibaba Cloud internal DNS servers mentioned above for resolution.

Step 1: Enable internal domain name resolution for the API Server

Call the ModifyCluster OpenAPI operation to create an internal DNS record in the cluster's VPC. The steps in this section assume that your nodes use the default Alibaba Cloud internal DNS.

If your nodes or containers use a custom DNS server, see Custom DNS configurations.

Call the ModifyCluster operation and specify the following parameters.

{
    "control_plane_endpoints_config": {
        "internal_dns_config": {
            "enabled": true
        }
    }
}

During the update, the API Server automatically restarts. After the restart is complete, call the DescribeClusterDetail operation and check that the internal_dns_config.enabled parameter is set to true to confirm that the configuration has taken effect.

Step 2: Modify the kube-proxy configuration

After the API Server domain name takes effect, switch the kube-proxy connection address from the IP address to the domain name.

On the ACK Clusters page, click the name of the target cluster. In the left navigation pane, choose Configurations > ConfigMaps.
Set the Namespace to kube-system, and then locate kube-proxy-worker and kube-proxy-worker-windows.
For each kubeconfig field, update the server parameter to match the following format.
```
https://apiserver.{cluster_id}.{region_id}.cs.aliyuncs.com:6443
```
- {cluster_id}: The cluster ID.
- {region_id}: The region ID of the cluster, such as cn-hangzhou.

Step 3: Verify that the internal domain name is effective

Before you modify existing nodes, create a new node or pod to verify that the domain name resolution works for new resources.

Nodes in a node pool or Lingjun node pool:
Scale out a new node in the cluster. The kubelet and kube-proxy on the new node will automatically use the internal domain name.
After the node is ready, log on to the node and run the following command to check whether the server address in the kubelet configuration file is the internal domain name.
```
cat /etc/kubernetes/kubelet.conf
```
Virtual nodes:
ACK Virtual Node component must be version v2.14.0 or later.
After you enable the internal domain name, new ECI pods and ACS pods interact with the API Server through the internal domain name by default.
At this point, if you add a new pod, it should enter the Ready state.

Step 4: Recreate existing kube-proxy pods to apply the changes

After the kube-proxy configuration is updated, recreate the existing pods to load the new configuration.

On the ACK Clusters page, click the name of the target cluster. In the left navigation pane, choose Workloads > DaemonSets.
Set the Namespace to kube-system. Locate and click `kube-proxy-worker` and `kube-proxy-worker-windows`. On the Pods tab, delete their pods one by one. This action recreates the pods and loads the new configuration.
Important
To ensure service stability, delete the pods in batches in a grayscale manner. Verify that the pods are recreated successfully before you delete the next batch.

Step 5: Modify the kubelet configuration on existing nodes

Update the kubelet configuration on existing nodes and restart the service to complete the switch to internal domain name access. To ensure cluster stability, this section is divided into two parts: single-node verification and batch operations.

Verify on a single node.
1. Select an existing node. Log on to the node and run the following command to switch the API Server address from an IP address to a domain name and restart the kubelet.
  - {clb-ip}: The IP address of the CLB configured in the /etc/kubernetes/kubelet.conf file.
  - {cluster-id}: The cluster ID
  - {region-id}: The region ID
```
# The following one-line command replaces the configuration and restarts the service.
sed -i.bak 's#https://{slb-ip}:6443#https://apiserver.{cluster-id}.{region-id}.cs.aliyuncs.com:6443#g' /etc/kubernetes/kubelet.conf && systemctl restart kubelet
```
  Example:
```
sed -i.bak 's#https://10.XX.XX.1:6443#https://apiserver.c34f3cdb665954e2f8e8e40a824a*****.cn-hangzhou.cs.aliyuncs.com:6443#g' /etc/kubernetes/kubelet.conf && systemctl restart kubelet
```
2. After the command is executed, monitor the node in the console to ensure that it is in the Ready state.
Perform batch operations on the remaining nodes.
After the verification on a single node is successful, use the batch operations feature in the console to perform batch operations on the remaining existing nodes.
1. On the ACK Clusters page, click the name of the target cluster. In the left navigation pane, choose Nodes > Nodes.
2. Select the nodes on which you want to perform the operation. At the bottom of the page, click Batch Operations and then select Execute Shell Command. Configure the command as prompted and submit the batch task.
```
# Enter the following command.
sed -i.bak 's#https://{clb-ip}:6443#https://apiserver.{cluster-id}.{region-id}.cs.aliyuncs.com:6443#g' /etc/kubernetes/kubelet.conf && systemctl restart kubelet
```
3. Wait for the task to complete. After you confirm that the node status is normal, repeat this operation until all existing nodes are updated.

Appendix

Custom DNS configurations

If your nodes or containers are configured with a custom DNS server, see this section to enable internal domain name resolution for the API Server.

The API Server domain name resolution depends on the default internal DNS service addresses 100.100.2.136 and 100.100.2.138 configured in the VPC. If a custom DNS server needs to resolve this domain name, you must associate the VPC where the DNS server is located with the private zone used for internal domain name resolution.

Procedure

Call the ModifyCluster OpenAPI operation to modify the cluster configuration. When you enable the API Server access optimization, use the bind_vpcs parameter to associate the VPC of the custom DNS server.

This step assumes that the VPC that hosts the DNS server and the cluster are under the same account. If they belong to different accounts, see Associate a VPC across accounts to centrally manage internal DNS resolution (from Step 1 to 5).

Call the ModifyCluster OpenAPI operation and specify the following content.
In ["vpc-id"], enter the VPC ID of the DNS server. This is an array. Separate multiple VPC IDs with commas (,).
```
{
    "control_plane_endpoints_config": {
        "internal_dns_config": {
            "enabled": true,
            "bind_vpcs": ["vpc-id"] 
        }
    }
}
```
After the configuration takes effect, you can call the DescribeClusterDetail operation to verify the configuration.
What to do next:
- If this is the first time you are enabling internal domain name access, continue with Step 2 through Step 5 to complete the subsequent configuration updates and verification.
- If you have already enabled internal domain name access and are only adding an associated VPC (bind_vpcs), the operation is complete.

Container Service for Kubernetes:Enable internal domain name access to the API Server for an existing cluster