Setting a Whitelist to Specified Pod Segments in Terway for ACK
By Daina
Generally, a whitelist for services, such as databases, is configured to provide services with safer access control capabilities. The scenario in the container network has the same demand. How can we set a whitelist for dynamic Pod IPs in container scenarios?
The container network of Alibaba Cloud Container Service for Kubernetes (ACK) mainly includes Flannel and Terway. In Flannel, the access to other services of a Pod is implemented through the node NAT. Thus, when setting the whitelist to a database in Flannel, we can schedule the client Pod to a small number of fixed nodes by binding nodes. Then, on the database side, the IP address of the node can be whitelisted. This method will not be described in detail. However, in Terway, the Pod IP is provided through EtherCAT Network Information (ENI). The client IP obtained by the external service is the IP address provided by ENI. With an affinity binding between Pods and nodes, the client IP of the Pod accessing external service is the IP provided by ENI, not the nodes. A Pod IP also randomly allocates IP addresses from the vSwitch specified by Terway. Besides, the client Pods are usually configured with automatic scaling. Thus, it is still difficult to enable auto-scaling scenarios even with a fixed Pod IP. The best way is to appoint a network segment directly to the client to allocate IP and then set whitelists to the segment in the database. However, how can we specify the IP range of a Pod in Terway?
Terway allows Pods on specified nodes to use the corresponding vSwitch. For more information, please see this link (article in Chinese):
Simply speaking, it specifies the vSwitch used by the Pod by adding labels to the specified node. Therefore, when scheduled a Pod to the node with a fixed label, the Pod can use its customized vSwitch to create a Pod IP. The related steps are as follows:
- Create an individual
configmap eni-config-fixedinstance in kube-system and specify vSwitch, vsw-2zem796p76viir02c6980 10.2.1.0/24.
apiVersion: v1
data:
eni_conf: |
{
"vswitches": {"cn-beijing-h":["vsw-2zem796p76viir02c6980"]}
}
kind: ConfigMap
metadata:
name: eni-config-fixed
namespace: kube-system
- Create a node pool and label the nodes with terway-config:
eni-config-fixed. To avert other Pods on the nodes in the node pool, label the node pool with taints, for example,fixed=true:NoSchedule. - Nodes added by the node pool contain the taints and labels above by default.
- Create a Pod and schedule it to the labeled node. It is advisable to add tolerations.
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
name: nginx-fixed
labels:
app: nginx-fixed
spec:
replicas: 2
selector:
matchLabels:
app: nginx-fixed
template:
metadata:
labels:
app: nginx-fixed
spec:
tolerations:
- key: "fixed"
operator: "Equal"
value: "true"
effect: "NoSchedule"
nodeSelector:
terway-config: eni-config-fixed
containers:
- name: nginx
image: nginx:1.9.0 # replace it with your exactly <image_name:tags>
ports:
- containerPort: 80
The Pod IP has been assigned from the specified vSwitch.
kubectl get po -o wide | grep fixed
nginx-fixed-57d4c9bd97-lwfxr 1/1 Running 0 39s 10.2.1.124 bj-tw.062149.aliyun.com <none> <none>
nginx-fixed-57d4c9bd97-tk4j9 1/1 Running 0 39s 10.2.1.125 bj-tw.062148.aliyun.com <none> <none>
- After scaling the deployment, all the generated Pod IPs are under the specified vSwitch.
kubectl scale deployment nginx-fixed --replicas=30
nginx-fixed-57d4c9bd97-2rxjz 1/1 Running 0 60s 10.2.1.132 bj-tw.062148.aliyun.com <none> <none>
nginx-fixed-57d4c9bd97-4w76t 1/1 Running 0 60s 10.2.1.144 bj-tw.062149.aliyun.com <none> <none>
nginx-fixed-57d4c9bd97-5brzw 1/1 Running 0 60s 10.2.1.143 bj-tw.062148.aliyun.com <none> <none>
...
- This way, we can whitelist the vSwitch directly on the database side to implement access control for dynamic Pod IPs.
Notes:
- The best way is to create a new node. For existing nodes, you must unbind the ENI from the Alibaba Cloud Elastic Compute Service (ECS) instance before adding them to the cluster, which can be performed on the ECS console. Then, you can add the existing nodes automatically to the cluster to replace the system disk.
- Add the labels and taints to the specific node pool to ensure that the business that does not need to be whitelisted is not scheduled to these nodes.
- This operation is a configuration overlay that overwrites the previous configuration of eni-config with that in the specified configmap. Please refer to this link (article in Chinese) to configure.
- It’s recommended that the number of IP addresses in the specified vSwitch is twice or more than the expected number of Pods. This leaves more space for later expansions and avoids the situation where no IP can be assigned when the Pod IPs cannot be recovered in time due to the occurrence of the fault.
Read previous post:
The Service Discovery Principle of DNS in Kubernetes Clusters
Read next post:
-
-
Alibaba Container Service
230 posts | 34 followers
Follow
Comments
-
-
Alibaba Container Service
230 posts | 34 followers
Follow
Related Products
-
Container Service for Kubernetes
Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.
Learn More -
ACK One
Provides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resources
Learn More -
Container Compute Service (ACS)
A cloud computing service that provides container compute resources that comply with the container specifications of Kubernetes
Learn More -
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn More
More Posts by Alibaba Container Service
- ACK One Fleet: Scaling Agent Sandboxes from Single-Cluster to Multi-Cluster
- Deep Dive: How Kimi's AI Agent Runs on Alibaba Cloud
- Caching is Efficiency: Achieving Precise LLM Cache Hits with Alibaba Cloud ACK GIE
- ACK One Fleet Multi-Cluster Canary Release: A "Safety Valve" for AI Inference Services
- Intelligent Scheduling for AI Inference: Cluster-Level Priority Elastic Scheduling
- When Agents Meet Workflows—Can Intelligence Become More Controllable?
- Koordinator Column 1: Viewing AI Computing Power's "Rigidity" and "Elasticity" through Gang Scheduling
- Container Technology Evolution for LLMs and AI Agents
- How ASM Ambient Mode Innovates Kubernetes Egress Traffic Management
- Best Practices for Ray on ACK: Secure Deployment of AI Data Processing/Training/Inference Environments