Container Compute Service:Enhanced egress traffic for ACS Agent Sandbox

Step 1: Create a Sandbox

To create a Sandbox, use the network.alibabacloud.com/network-policy-mode annotation to specify a network policy mode. This example creates a Sandbox with the Enhanced Traffic Policy mode enabled, which automatically injects a traffic-proxy sidecar.

1. Save the following content as sandbox.yaml and run kubectl apply -f sandbox.yaml:

   apiVersion: agents.kruise.io/v1alpha1
   kind: Sandbox
   metadata:
     name: sample
     namespace: default
   spec:
     template:
       metadata:
         labels:
           agent: sample
           alibabacloud.com/compute-class: agent-sandbox
         annotations:
           network.alibabacloud.com/network-policy-mode: enhanced-traffic-policy
       spec:
         containers:
         - name: sandbox
           image: registry.cn-hangzhou.aliyuncs.com/acs/curl:8.1.2
           command: ["/bin/sleep", "infinity"]

   Key configuration

   • network.alibabacloud.com/network-policy-mode: This annotation specifies the network policy mode for the Sandbox. For the differences between the values, see Overview.

   • spec.template.metadata.labels: The pod's labels. A SecurityProfile uses its selector to match these labels and apply security policies.

2. Run the following command to verify that the Sandbox is running:

   kubectl get sandbox sample -n default

   Expected output:

   NAME     STATUS    AGE
   sample   Running   2m

3. Run the following command to verify that the traffic-proxy sidecar was injected:

   kubectl get pod -l agent=sample -n default

   Expected output:

   NAME      READY   STATUS    RESTARTS   AGE
   sample    2/2     Running   0          2m

   READY count of 2/2 indicates that both the application container and the traffic-proxy sidecar are ready.

Step 2: Verify default behavior

If no SecurityProfile is configured, all egress traffic from the Sandbox is allowed by default.

1. Test access to an external service:

   kubectl exec sample -n default -c sandbox -- curl -s http://httpbin.org/get

   An HTTP 200 response is returned:

   {
     "args": {},
     "headers": {
       "Host": "httpbin.org",
       "User-Agent": "curl/8.1.2"
     },
     "origin": "47.239.x.x",
     "url": "http://httpbin.org/get"
   }

2. Test access to the /admin path:

   kubectl exec sample -n default -c sandbox -- curl -s -o /dev/null -w "%{http_code}" http://httpbin.org/admin

   The expected output is 404. This status code from httpbin.org indicates that the request reached the target service and was not blocked.

Step 3: Create a block rule

Create a SecurityProfile to block access to paths prefixed with /admin on any domain. For a complete example and configuration details, see Complete example.

1. Save the following content as securityprofile.yaml and run kubectl apply -f securityprofile.yaml:

   apiVersion: agents.kruise.io/v1alpha1
   kind: SecurityProfile
   metadata:
     name: agent-profile
     namespace: default
   spec:
     selector:
       matchLabels:
         agent: sample
     rules:
     - name: deny-admin
       match:
       - domains:
         - "*"
         paths:
         - type: Prefix
           value: /admin
         methods:
         - GET
         - POST
         - PUT
         - DELETE
       actions:
         block:
           statusCode: 403
           body: '{"error":"admin path is blocked"}'

2. Check the status of the SecurityProfile:

   kubectl get securityprofile -n default

   Expected output:

   NAME            AGE
   agent-profile   30s

Step 4: Verify traffic blocking

The SecurityProfile rules take effect immediately upon creation.

• Test 1: Access the blocked /admin path

  kubectl exec sample -n default -c sandbox -- curl -s http://httpbin.org/admin

  Expected output:

  {"error":"admin path is blocked"}

  Verify the status code:

  kubectl exec sample -n default -c sandbox -- curl -s -o /dev/null -w "%{http_code}" http://httpbin.org/admin

  The expected output is 403.

• Test 2: Verify that /admin subpaths are also blocked

  Because the rule uses Prefix matching, all paths that start with /admin are blocked:

  kubectl exec sample -n default -c sandbox -- curl -s http://httpbin.org/admin/users

  Expected output:

  {"error":"admin path is blocked"}

• Test 3: Verify that non-/admin paths are allowed

  kubectl exec sample -n default -c sandbox -- curl -s -o /dev/null -w "%{http_code}" http://httpbin.org/get

  The expected output is 200.

• Test 4: Verify that methods not in the list are unaffected

  kubectl exec sample -n default -c sandbox -- curl -s -o /dev/null -w "%{http_code}" -X OPTIONS http://httpbin.org/admin

  The rule applies only to GET, POST, PUT, and DELETE methods. Since the OPTIONS method is not included, the proxy allows the request.

Step 5: Update SecurityProfile (Optional)

You can update SecurityProfile rules at any time, and the changes take effect immediately. The following example adds a rule to block access to a specific domain:

apiVersion: agents.kruise.io/v1alpha1
kind: SecurityProfile
metadata:
  name: agent-profile
  namespace: default
spec:
  selector:
    matchLabels:
      agent: sample
  rules:
  - name: deny-admin
    match:
    - domains:
      - "*"
      paths:
      - type: Prefix
        value: /admin
      methods:
      - GET
      - POST
      - PUT
      - DELETE
    actions:
      block:
        statusCode: 403
        body: '{"error":"admin path is blocked"}'
  - name: deny-internal-service
    match:
    - domains:
      - "*.internal.company.com"  # The domain to block. Wildcards are supported.
    actions:
      block:
        statusCode: 403                                               # custom response status code
        body: '{"error":"access to internal services is forbidden"}'  # custom response body

Step 6: Delete SecurityProfile (Optional)

kubectl delete securityprofile agent-profile -n default

Scenario 1: Block admin paths

Prevent an agent from accessing administrative paths like /admin, /console, and /dashboard on any domain.

rules:
- name: deny-management-paths
  match:
  - domains: ["*"]
    paths:
    - type: Prefix
      value: /admin
    - type: Prefix
      value: /console
    - type: Prefix
      value: /dashboard
    - type: Prefix
      value: /manage
  actions:
    block:
      statusCode: 403
      body: '{"error":"management paths are blocked"}'

Scenario 2: Block sensitive domains and metadata services

Block access to internal service domains and cloud platform metadata service endpoints to mitigate SSRF attacks.

rules:
- name: deny-internal-and-metadata
  match:
  - domains:
    - "*.internal.company.com"
    - "*.corp.net"
    - "metadata.google.internal"
  actions:
    block:
      statusCode: 403
      body: '{"error":"access to internal services and metadata endpoints is forbidden"}'

Scenario 3: Read-only mode

Allow only GET/HEAD requests and block all write operations. This is useful for agents that require only read access.

rules:
- name: read-only-mode
  match:
  - domains: ["*"]
    paths:
    - type: Prefix
      value: /
    methods: ["POST", "PUT", "DELETE", "PATCH"]
  actions:
    block:
      statusCode: 405
      body: '{"error":"write operations are not allowed in read-only mode"}'

Scenario 4: Block a specific API endpoint

Use the Exact match type to block a specific API path without affecting other endpoints that share the same prefix.

rules:
- name: deny-user-deletion
  match:
  - domains: ["api.example.com"]
    paths:
    - type: Exact
      value: /api/v1/users/delete
    methods: ["POST", "DELETE"]
  actions:
    block:
      statusCode: 403
      body: '{"error":"user deletion API is not permitted"}'

Scenario 5: Block dynamic paths with regex

Use the Regex match type to block dynamic paths that match a specific pattern, such as /internal/v1/... and /internal/v2/....

rules:
- name: deny-internal-versioned-api
  match:
  - domains: ["*"]
    paths:
    - type: Regex
      value: "^/internal/v[0-9]+/.*"
  actions:
    block:
      statusCode: 403
      body: '{"error":"access to internal versioned APIs is blocked"}'

Scenario 6: Block by request header

Prevent an agent from impersonating an administrator by sending a forged request header to an external service.

rules:
- name: deny-admin-role-header
  match:
  - domains: ["*"]
    headers:
    - name: X-User-Role
      type: Exact
      value: admin
  actions:
    block:
      statusCode: 403
      body: '{"error":"requests with admin role header are not allowed"}'

Scenario 7: Fine-grained rule with multiple conditions

Within a single match item, fields are joined by a logical AND; all conditions must be met. Multiple match items are joined by a logical OR; the rule triggers if any item's conditions are met.

rules:
- name: deny-json-upload
  match:
  - domains: ["storage.example.com"]
    paths:
    - type: Prefix
      value: /upload
    methods: ["POST", "PUT"]
    headers:
    - name: Content-Type
      type: Prefix
      value: "application/json"
  actions:
    block:
      statusCode: 403
      body: '{"error":"JSON upload to storage service is not allowed"}'

Scenario 8: Block non-standard ports

Block attempts to use non-standard ports to evade security audits or perform unauthorized access.

rules:
- name: deny-non-standard-ports
  match:
  - domains: ["*"]
    ports: [8080, 8443, 9090, 3000, 6379, 27017]
  actions:
    block:
      statusCode: 403
      body: '{"error":"access to non-standard ports is blocked"}'

Example

The following example combines multiple matching criteria, such as domain name wildcards, path matching (prefix, exact, and regular expression), and HTTP method filtering:

apiVersion: agents.kruise.io/v1alpha1
kind: SecurityProfile
metadata:
  name: comprehensive-profile
  namespace: default
spec:
  selector:
    matchLabels:
      app: ai-agent
  rules:
  # Block all administrative paths
  - name: deny-admin-paths
    match:
    - domains: ["*"]
      paths:
      - type: Prefix
        value: "/admin"
      - type: Prefix
        value: "/console"
      - type: Prefix
        value: "/dashboard"
      methods: ["GET", "POST", "PUT", "DELETE"]
    actions:
      block:
        statusCode: 403
        body: '{"error":"management paths are blocked"}'
  # Read-only mode - Block all write operations
  - name: read-only-mode
    match:
    - domains: ["*"]
      paths:
      - type: Prefix
        value: "/"
      methods: ["POST", "PUT", "DELETE", "PATCH"]
    actions:
      block:
        statusCode: 405
        body: '{"error":"write operations are not allowed"}'
  # Block internal versioned APIs using a regular expression
  - name: deny-internal-api
    match:
    - domains: ["api.internal.com"]
      paths:
      - type: Regex
        value: "^/api/v[0-9]+/(users|tokens)/delete"
    actions:
      block:
        statusCode: 403
        body: '{"error":"deletion APIs are blocked"}'
  # Block non-standard ports
  - name: deny-debug-ports
    match:
    - domains: ["*"]
      ports: [8080, 9090, 3000]
    actions:
      block:
        statusCode: 403
        body: '{"error":"non-standard port access is blocked"}'

SecurityProfileSpec

The spec defines the Layer 7 (L7) security policies that apply to the egress traffic of target Pods.

SecurityRule

A single rule in the rule chain. It uses default-continue semantics: After a rule's action is executed, evaluation proceeds to the next rule, unless a terminating action such as block short-circuits the evaluation.

RuleMatch

A single match condition. Multiple fields within a single RuleMatch are combined with a logical AND. Multiple RuleMatch items are combined with a logical OR.

SecurityRuleActions

The action configuration. In the current version, only the block action is supported. Support for other action types is planned for future releases.

Component logs

kubectl logs -l gateway.networking.k8s.io/gateway-name=egress-gateway -n sandbox-traffic-system -f

kubectl logs -l app=gateway-controller -n sandbox-traffic-system -f

Metrics

The components expose the following Prometheus metrics.

The components are pre-configured with the prometheus.io/scrape: "true" Pod annotation. If your cluster has Alibaba Cloud Prometheus Monitoring enabled or has community-edition Prometheus deployed with annotation-based service discovery rules, metrics are collected automatically.

Default component specifications

The enhanced egress traffic management components are deployed in the sandbox-traffic-system namespace. The default specifications are as follows:

egress-gateway and traffic-extension

The capacity of these two components is limited by the CPU and network bandwidth of the underlying compute resources. We recommend using the same specifications for both.

• High QPS scenarios (small requests)

  QPS capacity of a single egress-gateway pod (8C 8Gi):

  Compute type	QPS capacity per pod
  ACS general-purpose	Approximately 10,000
  ACS performance	Approximately 25,000

  Note

  This data is from lab tests and is not a performance guarantee. To evaluate actual performance, monitor for packet loss under your workload or check for502 status codes in the gateway logs.

  Limitations:

  • For high QPS scenarios, we recommend that you select ACS performance pods.

  • A single pod's QPS is capped by the Linux ephemeral port limit (default: 32768–60999). To handle higher traffic volumes, use HPA for horizontal scaling to add more replicas.

• Large file transfer scenarios (high bandwidth)

  The throughput of the egress-gateway primarily depends on the network bandwidth limit of the underlying compute resources:

  Compute type	Maximum network bandwidth per pod
  ACS general-purpose (8C)	Approximately 10.5 Gbps
  ACS performance (8C)	Approximately 16 Gbps

  Note

  The maximum network bandwidth for an ACS pod is determined by the specifications in ACS instance types.

  Limitations:

  • Latency increases significantly as the network bandwidth approaches its limit. We recommend that you monitor the network throughput.

  • For ACS, you can distribute traffic across multiple pods by increasing the number of replicas.

  • For large file transfer scenarios, we recommend that you allocate at least 8Gi of memory to the egress-gateway.

gateway-controller

Resource requirements depend on the number of pods managed in the cluster:

Adjust egress-gateway or traffic-extension resources

Perform this operation during off-peak hours.

1. On the details page of the target ACK cluster, choose Add-ons in the navigation pane on the left.

2. On the Add-ons page, search for ack-sandbox-manager and then click Configuration.

3. On the component's Configuration page, adjust the resources value for the egress-gateway or traffic-extension component in the enhancedTrafficManagement section. Set the CPU requests and limits to the same value to ensure accurate autoscaling calculations based on CPU utilization. Then, click OK.