Container Compute Service:Manage agent network access with TrafficPolicy

TrafficPolicy and GlobalTrafficPolicy

and ACS clusters provide two types of network policy resources:

Policy evaluation rules

TrafficPolicy rules are evaluated based on the following principles:

1. Sorted by priority: The priority ranges from 1 to 1000. A smaller value indicates a higher priority. Policies with higher priority are evaluated first.

2. Sequential matching: Rules within the same policy are evaluated sequentially from top to bottom as they are defined.

3. First match wins: The first rule that matches is applied, and the evaluation stops.

4. Deny by default: If a policy selects a Pod, traffic to and from that Pod is denied by default unless explicitly allowed.

Supported match conditions

Ingress rules do not support FQDN matching.

Comparison with Kubernetes NetworkPolicy

TrafficPolicy and GlobalTrafficPolicy offer several advantages over the native Kubernetes NetworkPolicy:

Quick start

The following example shows how to configure a basic network policy for an agent application to allow access to the DNS Service and the public network.

apiVersion: v1
kind: Pod
metadata:
  name: agent-pod
  annotations:
    network.alibabacloud.com/enable-network-policy-agent: "true"
    network.alibabacloud.com/network-policy-mode: "traffic-policy"
  labels:
    app: agent
    alibabacloud.com/acs: "true"
spec:
  tolerations:
  - key: "virtual-kubelet.io/provider"
    operator: "Equal"
    value: "alibabacloud"
    effect: "NoSchedule"
  containers:
  - name: agent
    image: registry.aliyuncs.com/acs/ubuntu:latest
    command: ["sleep", "infinity"]

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: agent-basic-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: agent

  egress:
    rules:
      # Allow access to the DNS service.
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns

      # Allow access to the public network.
      - action: allow
        to:
          - cidr: 0.0.0.0/0

Allow access to specified services and domains

The following policy allows an agent to access the DNS Service, a specified domain, and a specified CIDR block, while denying access to the cloud metadata service and all other destinations.

Replace rds-xxx.aliyuncs.com with the actual domain name of your RDS instance.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: agent-egress-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: agent

  egress:
    rules:
      # Deny access to the cloud metadata service.
      - action: deny
        to:
          - cidr: 100.100.100.200/32

      # Allow access to a specified CIDR block.
      - action: allow
        to:
          - cidr: 100.96.0.0/10

      # Allow access to a specified domain name.
      - action: allow
        to:
          - fqdn: rds-xxx.aliyuncs.com

      # Allow access to the DNS service.
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns

      # Fallback rule: Deny access to all other destinations.
      - action: deny
        to:
          - cidr: 0.0.0.0/0

Deny access to private networks

This policy explicitly denies access to private network CIDR blocks while allowing access to the public network.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: agent-internet-only-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: agent
      
  egress:
    rules:
      # Allow DNS access.
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns
  
      # Deny access to private networks.
      - action: deny
        to:
          - cidr: 10.0.0.0/8
      - action: deny
        to:
          - cidr: 172.16.0.0/12
      - action: deny
        to:
          - cidr: 192.168.0.0/16
  
      # Allow access to the public network.
      - action: allow
        to:
          - cidr: 0.0.0.0/0

Allow access from a specified service

This policy allows only the Gateway Service to access an agent Pod.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: agent-ingress-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: agent

  ingress:
    rules:
      # Allow access from the Gateway service.
      - action: allow
        from:
          - service:
              name: sandbox-gateway
              namespace: sandbox-system

      # Fallback rule: Deny all other sources.
      - action: deny
        from:
          - cidr: 0.0.0.0/0

Cluster-wide access control

The following global policy denies a specific CIDR block from accessing Pods in the cluster, while allowing all other traffic.

apiVersion: network.alibabacloud.com/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: global-security-baseline
spec:
  priority: 1000    # Low priority, used as a fallback policy.
  selector: {}      # Applies to all matching Pods in the cluster.

  ingress:
    rules:
      # Deny access from the Sandbox CIDR block.
      - action: deny
        from:
          - cidr: 10.8.0.0/16

      # Allow all other sources.
      - action: allow
        from:
          - cidr: 0.0.0.0/0

  egress:
    rules:
      - action: allow
        to:
          - cidr: 0.0.0.0/0

Setting the selector in a GlobalTrafficPolicy to an empty {} applies it to all matching Pods across the entire cluster.

Combine multi-level security policies

You can combine multiple TrafficPolicy objects to implement layered control using different priority levels. The following table provides a recommended priority scheme and configuration examples.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: infra-policy
  namespace: default
spec:
  priority: 10
  selector:
    matchLabels:
      app: agent
  egress:
    rules:
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns
      - action: allow
        to:
          - service:
              namespace: monitoring
              name: prometheus

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: business-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: agent
  egress:
    rules:
      - action: allow
        to:
          - service:
              namespace: default
              name: redis
      - action: allow
        to:
          - service:
              namespace: default
              name: mysql

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: internet-policy
  namespace: default
spec:
  priority: 500
  selector:
    matchLabels:
      app: agent
  egress:
    rules:
      - action: allow
        to:
          - cidr: 0.0.0.0/0

Complete network policy for OpenClaw

The following example shows a complete network isolation policy for the OpenClaw application, which includes both ingress and egress rules.

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: openclaw-policy
  namespace: default
spec:
  priority: 100
  selector:
    matchLabels:
      app: openclaw

  ingress:
    rules:
      # Allow access from Sandbox Gateway and Sandbox Manager.
      - action: allow
        from:
          - service:
              name: sandbox-gateway
              namespace: sandbox-system
          - service:
              name: sandbox-manager
              namespace: sandbox-system
      # Deny all other sources.
      - action: deny
        from:
          - cidr: 0.0.0.0/0

  egress:
    rules:
      # Deny access to the cloud metadata service.
      - action: deny
        to:
          - cidr: 100.100.100.200/32

      # Allow access to the DNS service.
      - action: allow
        to:
          - service:
              namespace: kube-system
              name: kube-dns

      # Allow access to the LLM proxy service.
      - action: allow
        to:
          - service:
              namespace: llm-proxy
              name: llm-proxy

      # Deny access to private networks.
      - action: deny
        to:
          - cidr: 10.0.0.0/8
      - action: deny
        to:
          - cidr: 172.16.0.0/12
      - action: deny
        to:
          - cidr: 192.168.0.0/16

      # Allow access to the public network.
      - action: allow
        to:
          - cidr: 0.0.0.0/0

Principle of least privilege

Grant only the necessary network permissions to Pods. Explicitly list the services and addresses that are allowed, and deny all other traffic by default.

egress:
  rules:
    # Explicitly allow required services.
    - action: allow
      to:
        - service:
            namespace: kube-system
            name: kube-dns
    - action: allow
      to:
        - fqdn: api.aliyun.com

    # Deny access to private networks.
    - action: deny
      to:
        - cidr: 10.0.0.0/8

    # Allow public network access as needed.
    - action: allow
      to:
        - cidr: 0.0.0.0/0

Design precise label selectors

Use multi-dimensional labels to precisely target Pods and prevent policies from unintentionally affecting other workloads.

# Recommended: Multi-dimensional labels
selector:
  matchLabels:
    app: agent
    environment: production
    team: ai-platform

# Avoid: Overly broad labels
selector:
  matchLabels:
    app: agent

Label design recommendations:

• app: Identifies the application type.

• environment: Distinguishes between environments (for example, production, staging, and development).

• team: Identifies the owner team.

Combine TrafficPolicy with GlobalTrafficPolicy

Use a GlobalTrafficPolicy as a cluster-wide security baseline and use a TrafficPolicy for fine-grained, application-level control.

Regularly audit policies

Regularly audit your policies:

• Check whether TrafficPolicy objects meet the latest security requirements.

• Review logs of denied connections to identify potential issues.

• Clean up policies that are no longer in use.

• Adjust policies based on business changes.

• Verify that priority settings are reasonable.

Example audit script:

#!/bin/bash

echo "=== All TrafficPolicies ==="
kubectl get trafficpolicy --all-namespaces -o wide

echo -e "\n=== All GlobalTrafficPolicies ==="
kubectl get globaltrafficpolicy -o wide

echo -e "\n=== Agent Pod List ==="
kubectl get pods -l app=agent --all-namespaces

Manage policies with version control

Store your TrafficPolicy configurations in a Git repository and organize them into directories by purpose.

traffic-policies/
├── base/                          # Base policies
│   ├── global-security-baseline.yaml
│   └── dns-access.yaml
├── applications/                  # Application policies
│   ├── openclaw-policy.yaml
│   └── agent-policy.yaml
└── environments/                  # Environment-specific differences
    ├── production/
    │   └── overrides.yaml
    └── staging/
        └── overrides.yaml

Policy does not take effect

Symptom: A TrafficPolicy is created, but network access is not restricted as expected.

Troubleshooting steps:

1. Check the Poseidon component status. Confirm that it is installed, the version is v0.7.0 or later, and the TrafficPolicy feature is enabled.

2. Check the Pod annotations. Confirm that the Pod is configured with the required annotations.

3. Check the Pod labels. Confirm that the Pod labels exactly match the selector.matchLabels of the policy.

   kubectl get pod <pod-name> --show-labels

4. Check the rule order. Confirm that specific rules are placed before any fallback rule.

   kubectl get trafficpolicy <policy-name> -o yaml

Policy conflicts

Symptom: Multiple policies apply to the same Pod, and the network behavior is not as expected.

Troubleshooting steps:

1. List all policies and their priorities:

   kubectl get trafficpolicy,globaltrafficpolicy -A -o wide

2. Check whether policies with the same priority apply to the same Pod.

3. Adjust the priority value. Use a smaller value for a higher priority.

4. A strategy to merge conflicts and avoid duplicate definitions.

Network connection timeout

Symptom: A Pod cannot access an expected address or service.

Troubleshooting steps:

1. Check if the policy is missing necessary allow rules, such as for the DNS service.

2. Check if the fallback rule is too strict and blocks necessary network access.

3. Add the necessary allow rule to the policy:

   egress:
     rules:
       - action: allow
         to:
           - service:
               namespace: <namespace>
               name: <service-name>

4. Ensure that the allow rule has a higher priority than the deny rule, or place the allow rule before the deny rule within the same policy.

Pod is inaccessible

Symptom: Other services cannot access the target Pod.

Troubleshooting steps:

1. Check that the Ingress rules allow the required source services.

2. Add an allow rule for the source service:

   ingress:
     rules:
       - action: allow
         from:
           - service:
               name: <source-service>
               namespace: <source-namespace>

TrafficPolicy field descriptions

apiVersion: network.alibabacloud.com/v1alpha1
kind: TrafficPolicy
metadata:
  name: <policy-name>        # Policy name. Must be unique within the namespace.
  namespace: <namespace>     # Namespace where the policy is located.
spec:
  priority: <1-1000>         # Priority. A smaller value indicates a higher priority.
  selector:                  # Pod selector.
    matchLabels:
      <label-key>: <label-value>

  ingress:                   # Ingress rules (optional).
    rules:
      - action: allow|deny
        from:
          - cidr: <cidr>
          - service:
              name: <name>
              namespace: <namespace>

  egress:                    # Egress rules (optional).
    rules:
      - action: allow|deny
        to:
          - cidr: <cidr>
          - service:
              name: <name>
              namespace: <namespace>
          - fqdn: <domain>

GlobalTrafficPolicy field descriptions

apiVersion: network.alibabacloud.com/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: <policy-name>        # Policy name. Must be unique within the cluster.
spec:
  priority: <1-1000>         # Priority. A smaller value indicates a higher priority.
  selector:                  # Pod selector. An empty {} selects all Pods in the cluster.
    matchLabels: {}

  ingress:                   # Ingress rules (optional).
    rules:
      - action: allow|deny
        from:
          - cidr: <cidr>
          - service:
              name: <name>
              namespace: <namespace>

  egress:                    # Egress rules (optional).
    rules:
      - action: allow|deny
        to:
          - cidr: <cidr>
          - service:
              name: <name>
              namespace: <namespace>
          - fqdn: <domain>

Matching condition reference