Use ASMEgressTrafficPolicy to manage egress traffic - Alibaba Cloud Service Mesh

Alibaba Cloud Service Mesh (ASM) instances of version 1.16.4 and later allow you to use CustomResourceDefinition (CRD) fields to define an egress traffic policy. This topic describes how to use an egress traffic policy to manage egress traffic.

Prerequisites

An ASM instance of a commercial edition (Enterprise Edition or Ultimate Edition) is created. The version of the instance is 1.16.4 or later. For more information about how to create an ASM instance, see Create an ASM instance. For more information about how to update an ASM instance, see Update an ASM instance.
Automatic sidecar proxy injection is enabled for the default namespace. For more information, see Enable automatic sidecar injection.

Feature description

ASM allows you to establish, manage, and protect communication between applications in a unified manner. ASM uses applications instead of IP addresses as the unit to which communication is allowed or denied, and you do not need to modify existing application code. An egress traffic policy defines how an egress gateway manages egress traffic. By using a combination of ASM egress gateways and authorization policies, you can control egress traffic in a flexible manner. Dingtalk_20230411112608

The traffic paths in this example can be divided into two types:

1. Communication between sidecar proxies and between sidecar proxies and gateways: By default, mutual Transport Layer Security (mTLS) is enabled for the communication, and certificates are managed by ASM.

2. Communication between applications and sidecar proxies and between gateways and external services:

a. To ensure that advanced Layer 7 capabilities of ASM can function properly, applications and sidecar proxies communicate in plaintext as much as possible. In this way, sidecar proxies can obtain Layer 7 information of the traffic and support more advanced features. Due to some special reasons, if applications must send HTTPS requests directly, only Layer 4 capabilities of ASM can be used.

b. You can configure the communication protocol between an egress gateway and an external service, either plaintext or HTTPS.

Preparations

1. Set an egress traffic policy for the ASM instance

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Sidecar Proxy Setting.
On the global tab, click Outbound Traffic Policy, set the Outbound Traffic Policy parameter to REGISTEY_ONLY, and then click Update Settings.

2. Create a namespace

Create a namespace named istio-egress. For more information, see Manage global namespaces.
On the Global Namespace page, click Sync Automatic Sidecar Injection to Kubernetes Cluster to synchronize the namespace to the ACK cluster that is added to the ASM instance.

3. Create an egress gateway

Create an egress gateway named egressgateway-a in the ASM console. In the Port Mapping section, add HTTP 80, HTTPS 443, and HTTPS 444, and select Support two-way TLS authentication for the egress gateway. For more information, see Create an egress gateway.

Make sure that the following content is included in the spec section in the generated YAML file of the egress gateway. If the YAML file does not contain the following content, add the content to the file.

spec:
  podLabels:
    security.istio.io/tlsMode: istio

4. Create services

Create a namespace named mytest for the ASM instance and enable automatic sidecar proxy injection. For more information, see Manage global namespaces.

In the ACK cluster, deploy the sleep-a service in the mytest namespace and deploy the nginx service in the default namespace.

Create the test.yaml file that contains the following content:

Expand to view the YAML file

apiVersion: v1
kind: Service
metadata:
  name: sleep-a
  namespace: mytest
  labels:
    app: sleep-a
    service: sleep-a
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep-a
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep-a
  namespace: mytest
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep-a
  template:
    metadata:
      labels:
        app: sleep-a
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: sleep
        image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: nginx
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/nginx:1.27.0
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

In the ACK cluster, run the following command to deploy the sleep-a and nginx services:
```
kubectl apply -f test.yaml
```

Run the following commands to use the sleep-a and nginx services to access http://www.httpbin.org:

kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

The return codes are both 502, which indicates that the access fails.

Use the sleep-a service to access an external service over HTTP

Solution 1: Use HTTP plaintext for communication between the sidecar proxy in the workload of the sleep-a service and the egress gateway

This solution is not applicable to actual environments. Currently, ASMEgressTrafficPolicy does not support using HTTP plaintext to access egress gateways.

Traffic management is implemented in sidecar proxies on the client side and does not require egress gateways.
Observability does not rely on egress gateways.
Security capabilities depend on egress gateways. However, without mTLS, all authorization capabilities based on client identities are unavailable. Egress gateways can only deny all requests indiscriminately.

(Recommended) Solution 2: Use mTLS for communication between the sidecar proxy and the egress gateway

Use the following content to create an egress-by-egressgateway.yaml file.

For more information about the fields, see CRD fields in an egress traffic policy.

apiVersion: istio.alibabacloud.com/v1
kind: ASMEgressTrafficPolicy
metadata:
  name: egress-by-egressgateway # The format of this value is egress-by-{Name of the egress gateway}. This value corresponds to the name of the egress gateway. 
  namespace: istio-egress         # This value is fixed as istio-egress. 
spec:
  byEgressGateway:
    name: egressgateway
  egressRules:
  - from:
    - namespace: mytest
      workloadSelector:
        app: sleep-a
    to:
    - name: httpbin-service-http
      hosts:
      - www.httpbin.org  # The IP addresses of multiple domain names after Domain Name System (DNS) resolution must be the same. 
      - httpbin.org      # The IP addresses of multiple domain names after DNS resolution must be the same. 
      port:
        name: http
        number: 80
        protocol: HTTP
      byEgressGateway:
        port: 80        # Traffic path: sidecar proxy > gateway (port 80) > httpbin.org service (port 80)

In the ACK cluster, run the following command to create an egress traffic policy:
```
kubectl apply -f egress-by-egressgateway.yaml
```
Check whether the egress traffic policy takes effect.
1. Run the following command to use the nginx service in the default namespace to access http://www.httpbin.org:
```
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
```
  The status code 502 is returned, which indicates that the nginx service fails to access http://www.httpbin.org.
2. Run the following command to use the sleep-a service in the mytest namespace to access http://www.httpbin.org:
```
kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org
```
  The status code 200 is returned, which indicates that the access succeeds. This meets expectations.
3. Run the following command to delete the egress-by-egressgateway egress traffic policy. Then, use the sleep-a service in the mytest namespace to access http://www.httpbin.org:
```
kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
```
  The status code 502 is returned, which indicates that the sleep-a service fails to access http://www.httpbin.org after the egress traffic policy is deleted.
The results indicate that the egress traffic policy takes effect.

Use the sleep-a service to access an external service over HTTPS

(Recommended) Solution 1: Use mTLS for communication between the sidecar proxy and the egress gateway

Use the following content to update the egress-by-egressgateway.yaml file.

Add the httpsUpgrade field and the definitions for directly accessing https://www.httpbin.org under the spec field. For more information about the fields, see CRD fields in an egress traffic policy.

apiVersion: istio.alibabacloud.com/v1
kind: ASMEgressTrafficPolicy
metadata:
  name: egress-by-egressgateway # The format of this value is egress-by-{Name of the egress gateway}. This value corresponds to the name of the egress gateway. 
  namespace: istio-egress          # This value is fixed as istio-egress. 
spec:
  byEgressGateway:
    name: egressgateway
  egressRules:
  - from:
     - namespace: mytest
       workloadSelector:
          app: sleep-a
    to:
    - name: httpbin-service-http
      hosts:
      - www.httpbin.org  # The IP addresses of multiple domain names after DNS resolution must be the same. 
      - httpbin.org      # The IP addresses of multiple domain names after DNS resolution must be the same. 
      port:
        name: http
        number: 80
        protocol: HTTP
      byEgressGateway:
        port: 80        # Traffic path: sidecar proxy > gateway (port 80) > httpbin.org service (port 80)
      httpsUpgrade:
        enabled: true # If this value is set to false, the value of the port parameter under httpsUpgrade does not take effect. 
        port: 443        # Traffic path: sidecar proxy > gateway (port 80) > httpbin.org service (port 443)

In the ACK cluster, run the following command to create an egress traffic policy:
```
kubectl apply -f egress-by-egressgateway.yaml
```
Check whether the egress traffic policy takes effect.
1. Check the sleep-a service in the mytest namespace.
  1. Run the following command to use the sleep-a service to access http://www.httpbin.org:
```
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin.org
```
    The status code 200 is returned, which indicates that the sleep-a service succeeds in accessing http://www.httpbin.org.
  2. Run the following command to use the sleep-a service to access httpbin.org over the anything port. Then, check whether the egress gateway upgrades HTTP requests to HTTPS requests.
```
kubectl -n mytest exec deployment/sleep-a -- sh -c "curl -s  http://httpbin.org/anything |grep url"
```
    Expected output:
```
"url": "https://httpbin.org/anything"
```
    In the output, the url field starts with https. This indicates that the egress gateway upgrades requests to HTTPS requests then forwards them to httpbin.org.
  3. Run the following command to use the sleep-a service in the mytest namespace to access https://www.httpbin.org:
```
kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org
```
    The status code 200 is returned, which indicates that the sleep-a service succeeds in accessing https://www.httpbin.org.
2. Check the nginx service in the default namespace.
  1. Run the following command to use the nginx service to access http://www.httpbin.org:
```
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
```
    The status code 502 is returned, which indicates that the nginx service fails to access http://www.httpbin.org.
  2. Run the following command to use the nginx service to access https://www.httpbin.org:
```
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" https://www.httpbin.org
```
    The output shows that the request is rejected, which indicates that the nginx service fails to access https://www.httpbin.org.
  3. Run the following command to view the access logs of the sidecar proxy in the nginx service workload and obtain the rejection reason:
```
 kubectl -n default  logs -f deployment/nginx -c istio-proxy   --tail=1 
```
    Expected output:
```
{"authority":"-","bytes_received":"0","bytes_sent":"0","downstream_local_address":"52.86.XX.XX:443","downstream_remote_address":"172.16.0.199:56748","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"-","request_id":"-","requested_server_name":"-","response_code":"0","response_flags":"UH","route_name":"-","start_time":"2023-04-11T02:00:07.409Z","trace_id":"-","upstream_cluster":"BlackHoleCluster","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}
```
    The output indicates that the request is forwarded to BlackHoleCluster, and therefore the request is rejected.
3. Run the following commands to delete the egress-by-egressgateway egress traffic policy and use the sleep-a service in the mytest namespace to access http://www.httpbin.org and https://www.httpbin.org again:
```
kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org
```
  When the service accesses http://www.httpbin.org, the status code 502 is returned. When the service accesses https://www.httpbin.org, the request is rejected. After the egress traffic policy is deleted, the sleep-a service cannot access http://www.httpbin.org or https://www.httpbin.org.
The output indicates that the egress traffic policy takes effect.

Solution 2: Use HTTPS for communication between the sidecar proxy and the egress gateway

Use the following content to update the egress-by-egressgateway.yaml file.

apiVersion: istio.alibabacloud.com/v1
kind: ASMEgressTrafficPolicy
metadata:
  name: egress-by-egressgateway # The format of this value is egress-by-{Name of the egress gateway}. This value corresponds to the name of the egress gateway. 
  namespace: istio-egress          # This value is fixed as istio-egress. 
spec:
  byEgressGateway:
    name: egressgateway
  egressRules:
  - from:
     - namespace: mytest
       workloadSelector:
          app: sleep-a
    to:
    - name: httpbin-service-https
      hosts:
      - www.httpbin.org
      - httpbin.org
      port:
        name: https
        number: 443
        protocol: HTTPS
      byEgressGateway:
        port: 444   # HTTPS port 444 that is configured in Step 3 in the Preparations section.

In the ACK cluster, run the following command to create an egress traffic policy:
```
kubectl apply -f egress-by-egressgateway.yaml
```
Check whether the egress traffic policy takes effect.
1. Check the sleep-a service in the mytest namespace.
  1. Run the following command to use the sleep-a service to access http://www.httpbin.org:
```
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin.org
```
    The status code 502 is returned, which indicates that the sleep-a service fails to access http://www.httpbin.org.
  2. Run the following command to use the sleep-a service in the mytest namespace to access https://www.httpbin.org:
```
kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org
```
    The status code 200 is returned, which indicates that the sleep-a service succeeds in accessing https://www.httpbin.org.
2. Check the nginx service in the default namespace.
  1. Run the following command to use the nginx service to access http://www.httpbin.org:
```
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
```
    The status code 502 is returned, which indicates that the nginx service fails to access http://www.httpbin.org.
  2. Run the following command to use the nginx service to access https://www.httpbin.org:
```
kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" https://www.httpbin.org
```
    The output shows that the request is rejected, which indicates that the nginx service fails to access https://www.httpbin.org.
  3. Run the following command to view the access logs of the sidecar proxy in the nginx service workload and obtain the rejection reason:
```
 kubectl -n default  logs -f deployment/nginx -c istio-proxy   --tail=1 
```
    Expected output:
```
{"authority":"-","bytes_received":"0","bytes_sent":"0","downstream_local_address":"52.86.XX.XX:443","downstream_remote_address":"172.16.0.199:56748","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"-","request_id":"-","requested_server_name":"-","response_code":"0","response_flags":"UH","route_name":"-","start_time":"2023-04-11T02:00:07.409Z","trace_id":"-","upstream_cluster":"BlackHoleCluster","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}
```
    The output indicates that the request is forwarded to BlackHoleCluster, and therefore the request is rejected.
3. Run the following commands to delete the egress-by-egressgateway egress traffic policy and use the sleep-a service in the mytest namespace to access http://www.httpbin.org and https://www.httpbin.org:
```
kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org
kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org
```
  When the service accesses http://www.httpbin.org, the status code 502 is returned. When the service accesses https://www.httpbin.org, the request is rejected. After the egress traffic policy is deleted, the sleep-a service cannot access http://www.httpbin.org or https://www.httpbin.org.
The results indicate that the egress traffic policy takes effect.

Related operations

Reject POST requests sent from a specified namespace

By using an egress gateway and an egress traffic policy, you can flexibly control outbound traffic in a cluster. In combination with an authorization policy, you can implement more fine-grained access control. For example, you can use the following policy to reject POST requests sent from the mytest namespace:

kind: AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
metadata:
  name: sleep-a-egress-www-httpbin-org
  namespace: istio-system
spec:
  action: DENY
  rules:
    - to:
        - operation:
            hosts:
            - www.httpbin.org
            - httpbin.org
            methods:
              - POST
      from:
        - source:
            namespaces: ["mytest"]
  selector:
    matchLabels:
      istio: egressgateway-a

After the preceding configuration is applied, RBAC: access is returned when POST requests are sent from the sleep-a service to access www.httpbin.org. GET requests to access www.httpbin.org are not affected.