Alibaba Cloud Service Mesh:Manage egress traffic with ASMEgressTrafficPolicy

How it works

A service mesh supports egress traffic management by creating and interconnecting multiple resources, such as ServiceEntry, VirtualService, Gateway, and Destination. Once these resources are configured, traffic is transparently intercepted and routed to an egress gateway, which then forwards the traffic to external services.

This configuration process is complex, error-prone, and requires a deep understanding of various fields. To simplify egress traffic configuration, ASM introduces the ASMEgressTrafficPolicy resource.

The ASMEgressTrafficPolicy resource abstracts and simplifies egress traffic configuration. Instead of manually creating and managing resources such as ServiceEntry, VirtualService, Gateway, and Destination, you only need to define a few essential settings. The service mesh then transparently intercepts traffic, routes it to an egress gateway, and forwards it to external services over HTTP or HTTPS.

Because ASMEgressTrafficPolicy is a simplified abstraction of native service mesh resources, it may not support advanced use cases, such as percentage-based traffic routing or initiating mTLS from the egress gateway. If you require advanced features or custom configurations, see Configure a unified egress gateway for in-mesh traffic.

Features

ASM provides a unified way to connect, manage, and secure communication between applications. Unlike network IP-based methods, ASM uses an application-centric approach that does not require changes to existing application code. The ASMEgressTrafficPolicy resource defines how to manage egress traffic through an egress gateway. By combining an ASM egress gateway with an AuthorizationPolicy, you can control egress traffic more flexibly.

This example includes the following traffic flows:

1. Communication between mesh proxies and between the proxies and the gateway: By default, ASM secures this communication with mTLS and manages the certificates.

2. Communication between applications and their mesh proxies, and between the gateway and external services:

a. To use the advanced L7 features of the service mesh, applications should communicate with their mesh proxies using plaintext whenever possible. This allows the mesh proxy to access L7 traffic information and support more advanced features. If an application must initiate HTTPS traffic directly, the service mesh can provide only L4 capabilities.

b. You can configure the communication protocol between the egress gateway and external services. Both plaintext and HTTPS are supported.

Set the outbound traffic policy

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Sidecar Proxy Setting.

3. On the global tab, click Outbound Traffic Policy. Then, to the right of Outbound Traffic Policy, click REGISTRY_ONLY and then click Update Settings.

Create a namespace

1. Create the istio-egress namespace. For more information, see Manage global namespaces.

2. On the Global Namespace page, click Sync Automatic Sidecar Injection to Kubernetes Cluster to synchronize the namespace with the ACK cluster managed by the ASM instance.

Create an egress gateway

In ASM, create an egress gateway named egressgateway-a. Under Port Mapping, configure HTTP on port 80 and HTTPS on ports 443 and 444. Then, enable Support two-way TLS authentication. For more information, see Create an egress gateway service.

spec:
  podLabels:
    security.istio.io/tlsMode: istio

Create a sample application

1. In the ASM instance, create a namespace named mytest and enable automatic sidecar injection. For more information, see Manage global namespaces.

2. In the ACK cluster, deploy the sleep-a service in the mytest namespace and the nginx service in the default namespace.

   1. Create a file named test.yaml with the following content:

      Test.yaml

      apiVersion: v1
      kind: Service
      metadata:
        name: sleep-a
        namespace: mytest
        labels:
          app: sleep-a
          service: sleep-a
      spec:
        ports:
        - port: 80
          name: http
        selector:
          app: sleep-a
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sleep-a
        namespace: mytest
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: sleep-a
        template:
          metadata:
            labels:
              app: sleep-a
          spec:
            terminationGracePeriodSeconds: 0
            containers:
            - name: sleep
              image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/curl:asm-sleep
              command: ["/bin/sleep", "infinity"]
              imagePullPolicy: IfNotPresent
              volumeMounts:
              - mountPath: /etc/sleep/tls
                name: secret-volume
            volumes:
            - name: secret-volume
              secret:
                secretName: sleep-secret
                optional: true
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app: nginx
        name: nginx
        namespace: default
      spec:
        progressDeadlineSeconds: 600
        replicas: 1
        revisionHistoryLimit: 10
        selector:
          matchLabels:
            app: nginx
        strategy:
          rollingUpdate:
            maxSurge: 25%
            maxUnavailable: 25%
          type: RollingUpdate
        template:
          metadata:
            creationTimestamp: null
            labels:
              app: nginx
          spec:
            containers:
            - image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/nginx:1.27.0
              imagePullPolicy: Always
              name: nginx
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30

   2. In the ACK cluster, run the following command to deploy the sleep-a and nginx services:

      kubectl apply -f test.yaml

3. Run the following commands to access http://www.httpbin.org from the sleep-a and nginx services:

   kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org
   kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

   The output for both commands is 502, which indicates that the access failed.

Option 1: Plaintext HTTP between sidecar and egress gateway

This option is not practical in a production environment. Plaintext communication between the application container and the egress gateway prevents client identity-based authorization. ASMEgressTrafficPolicy does not support accessing the egress gateway over plaintext HTTP.

• Traffic management is mainly enforced in the client sidecar and does not require an egress gateway.

• Observability does not depend on an egress gateway.

• Security features rely on the egress gateway. However, without mTLS, all authorization capabilities based on client identity are disabled. The egress gateway can only reject all requests.

Option 2: mTLS between sidecar and egress gateway (recommended)

1. Create an ASMEgressTrafficPolicy.

   Console

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Egress Gateway.

   3. Click the name of the gateway to go to the gateway overview page. In the left-side navigation pane, click Outbound Traffic Policy.

      Configure the egress traffic policy rule. Set Egress Gateway to egressgateway. In the In-cluster Service area, set Namespace to mytest, Workload Label Name to app, and Label Value to sleep-a. In the External Service area, set External Service Name to httpbin-service-http, add www.httpbin.org and httpbin.org to the Domain List. For Service Port, set Port to 80, Port Name to http, and Protocol to HTTP. In the How to Access area, select egress gateway port 80, and for Upgrade to HTTPS, select No.

   kubectl

   1. Create a file named egress-by-egressgateway.yaml with the following content.

      For descriptions of the fields, see ASMEgressTrafficPolicy CRD Reference.

      apiVersion: istio.alibabacloud.com/v1
      kind: ASMEgressTrafficPolicy
      metadata:
        name: egress-by-egressgateway  # Convention: egress-by-{egress-gateway-name}.
        namespace: istio-egress         # Convention: Fixed to the istio-egress namespace.
      spec:
        byEgressGateway:
          name: egressgateway
        egressRules:
        - from:
          - namespace: mytest
            workloadSelector:
              app: sleep-a
          to:
          - name: httpbin-service-http
            hosts:
            - www.httpbin.org  # All hosts must resolve to the same IP addresses.
            - httpbin.org      # All hosts must resolve to the same IP addresses.
            port:
              name: http
              number: 80
              protocol: HTTP
            byEgressGateway:
              port: 80        # Traffic path: sidecar -> Gateway (port 80) -> Service (httpbin.org, port 80)

   2. In your ACK cluster, run the following command to create the ASMEgressTrafficPolicy resource.

      kubectl apply -f egress-by-egressgateway.yaml

2. Verify that the ASMEgressTrafficPolicy takes effect.

   1. Run the following command to test access from the nginx service in the default namespace to http://www.httpbin.org.

      kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

      The command returns 502, which indicates that access from the nginx service to http://www.httpbin.org is blocked as expected.

   2. Run the following command to test access from the sleep-a service in the mytest namespace to http://www.httpbin.org.

      kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org

      The command returns 200, which indicates that access is allowed as expected.

   3. Run the following commands to delete the egress-by-egressgateway policy, and then test access from the sleep-a service in the mytest namespace to http://www.httpbin.org again.

      kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
      kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

      The command returns 502, which indicates that after the ASMEgressTrafficPolicy is deleted, the sleep-a service cannot access http://www.httpbin.org.

   These results confirm that the ASMEgressTrafficPolicy is working correctly.

Option 1: mTLS between sidecar and egress gateway (recommended)

1. Create an ASMEgressTrafficPolicy.

   Console

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Egress Gateway.

   3. Click the gateway name to open the gateway overview page. In the navigation pane on the left, click Outbound Traffic Policy.

      Configure an egress traffic rule. In the in-cluster service section, set Namespace to mytest, Label Name to app, and Label Value to sleep-a. In the external service section, set External Service Name to httpbin-service-http, add www.httpbin.org and httpbin.org to the Domain List, and configure the Service Port with Port: 80, Port Name: http, and Protocol: HTTP. In the How to Access section, set Egress Gateway Port to 80, Upgrade to HTTPS to Yes, and HTTPS Port to 443.

   Kubectl

   1. Update the egress-by-egressgateway.yaml file with the following content.

      The httpsUpgrade field and related definitions for accessing https://www.httpbin.org are added to the spec field. For descriptions of the fields, see ASMEgressTrafficPolicy CRD reference.

      apiVersion: istio.alibabacloud.com/v1
      kind: ASMEgressTrafficPolicy
      metadata:
        name: egress-by-egressgateway  # Convention: egress-by-{egress-gateway-name}.
        namespace: istio-egress          # Convention: Fixed to the istio-egress namespace.
      spec:
        byEgressGateway:
          name: egressgateway
        egressRules:
        - from:
           - namespace: mytest
             workloadSelector:
                app: sleep-a
          to:
          - name: httpbin-service-http
            hosts:
            - www.httpbin.org  # The IP addresses to which the multiple domain names resolve must be the same.
            - httpbin.org      # The IP addresses to which the multiple domain names resolve must be the same.
            port:
              name: http
              number: 80
              protocol: HTTP
            byEgressGateway:
              port: 80        # Sidecar -> Port 80 of Gateway -> Port 80 of Service (httpbin.org)
            httpsUpgrade:
              enabled: true   # If enabled is set to false, the port setting for httpsUpgrade has no effect.
              port: 443       # Sidecar -> Port 80 of Gateway -> Port 443 of Service (httpbin.org)

   2. In the ACK cluster, run the following command to create the ASMEgressTrafficPolicy resource:

      kubectl apply -f egress-by-egressgateway.yaml

2. Verify the ASMEgressTrafficPolicy configuration.

   1. Verify the sleep-a service in the mytest namespace.

      1. Run the following command to access http://www.httpbin.org from the sleep-a service.

         kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin.org

         A status code of 200 indicates that the sleep-a service can access http://www.httpbin.org.

      2. Run the following command to request the anything endpoint of httpbin.org and verify that the egress gateway upgrades the HTTP request to an HTTPS request.

         kubectl -n mytest exec deployment/sleep-a -- sh -c "curl -s  http://httpbin.org/anything |grep url"

         Expected output:

         "url": "https://httpbin.org/anything"

         The value of the url field starts with https, confirming that the egress gateway upgraded the request to https before forwarding it to httpbin.org.

      3. Run the following command to access https://www.httpbin.org from the sleep-a service in the mytest namespace.

         kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org

         A status code of 200 indicates that the sleep-a service can access https://www.httpbin.org.

   2. Verify the Nginx service in the default namespace.

      1. Run the following command to access http://www.httpbin.org from the Nginx service.

         kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

         A status code of 502 indicates that the Nginx service failed to access http://www.httpbin.org.

      2. Run the following command to access https://www.httpbin.org from the Nginx service.

         kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" https://www.httpbin.org

         A connection refused error indicates that the Nginx service failed to access https://www.httpbin.org.

      3. Run the following command to view the access log of the sidecar for the Nginx workload to identify why the connection was refused.

          kubectl -n default  logs -f deployment/nginx -c istio-proxy   --tail=1 

         Expected output:

         {"authority":"-","bytes_received":"0","bytes_sent":"0","downstream_local_address":"52.86.XX.XX:443","downstream_remote_address":"172.16.0.199:56748","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"-","request_id":"-","requested_server_name":"-","response_code":"0","response_flags":"UH","route_name":"-","start_time":"2023-04-11T02:00:07.409Z","trace_id":"-","upstream_cluster":"BlackHoleCluster","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}

         The output shows that the request is forwarded to the BlackHoleCluster, which rejects the connection.

   3. Run the following commands to delete egress-by-egressgateway, and then access http://www.httpbin.org and https://www.httpbin.org again from the sleep-a service in the mytest namespace.

      kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
      kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org
      kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org

      Requests to http://www.httpbin.org return 502, and requests to https://www.httpbin.org are rejected. After the ASMEgressTrafficPolicy configuration is deleted, the sleep-a service can no longer access http://www.httpbin.org or https://www.httpbin.org.

   These results confirm that the ASMEgressTrafficPolicy works as expected.

Option 2: HTTPS between the sidecar and the egress gateway

1. Create an ASMEgressTrafficPolicy.

   Console

   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.

   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Egress Gateway.

   3. Click the gateway name to open the gateway overview page. In the navigation pane on the left, click Outbound Traffic Policy.

      On the egress traffic rule page, click Add Rule and configure the rule parameters. Set Namespace to mytest. For Workload Labels, set the label name to app and the label value to sleep-a. Set External Service Name to httpbin-service-https. Add www.httpbin.org and httpbin.org to the Domain List. For Service Port, set the port to 443, port name to https, and Protocol to HTTPS. For How to Access, select Via egress gateway port and set the port to 444.

   Kubectl

   1. Update the egress-by-egressgateway.yaml file with the following content.

      The httpsUpgrade field and definitions for directly accessing https://www.httpbin.org are added to the spec field. For a description of the fields, see ASMEgressTrafficPolicy CRD.

      apiVersion: istio.alibabacloud.com/v1
      kind: ASMEgressTrafficPolicy
      metadata:
        name: egress-by-egressgateway  # Convention: egress-by-{egress-gateway-name}.
        namespace: istio-egress          # Convention: Fixed to the istio-egress namespace.
      spec:
        byEgressGateway:
          name: egressgateway
        egressRules:
        - from:
           - namespace: mytest
             workloadSelector:
                app: sleep-a
          to:
          - name: httpbin-service-https
            hosts:
            - www.httpbin.org
            - httpbin.org
            port:
              name: https
              number: 443
              protocol: HTTPS
            byEgressGateway:
              port: 444   # The HTTPS port 444 that is defined for the egress gateway in the prerequisites.

   2. In the ACK cluster, run the following command to create the ASMEgressTrafficPolicy resource:

      kubectl apply -f egress-by-egressgateway.yaml

2. Verify the ASMEgressTrafficPolicy configuration.

   1. Verify the sleep-a service in the mytest namespace.

      1. Run the following command to access http://www.httpbin.org from the sleep-a service.

         kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n" http://httpbin.org

         A status code of 502 indicates that the sleep-a service failed to access http://www.httpbin.org.

      2. Run the following command to access https://www.httpbin.org from the sleep-a service in the mytest namespace.

         kubectl -n mytest exec deployment/sleep-a -- curl  -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org

         A status code of 200 indicates that the sleep-a service can access https://www.httpbin.org.

   2. Verify the Nginx service in the default namespace.

      1. Run the following command to access http://www.httpbin.org from the Nginx service.

         kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" http://www.httpbin.org

         A status code of 502 indicates that the Nginx service failed to access http://www.httpbin.org.

      2. Run the following command to access https://www.httpbin.org from the Nginx service.

         kubectl -n default exec deployment/nginx -- curl -s -o /dev/null -w "%{http_code}\n" https://www.httpbin.org

         A connection refused error indicates that the Nginx service failed to access https://www.httpbin.org.

      3. Run the following command to view the access log of the sidecar for the Nginx workload to identify why the connection was refused.

          kubectl -n default  logs -f deployment/nginx -c istio-proxy   --tail=1 

         Expected output:

         {"authority":"-","bytes_received":"0","bytes_sent":"0","downstream_local_address":"52.86.XX.XX:443","downstream_remote_address":"172.16.0.199:56748","duration":"0","istio_policy_status":"-","method":"-","path":"-","protocol":"-","request_id":"-","requested_server_name":"-","response_code":"0","response_flags":"UH","route_name":"-","start_time":"2023-04-11T02:00:07.409Z","trace_id":"-","upstream_cluster":"BlackHoleCluster","upstream_host":"-","upstream_local_address":"-","upstream_service_time":"-","upstream_transport_failure_reason":"-","user_agent":"-","x_forwarded_for":"-"}

         The output shows that the request is forwarded to the BlackHoleCluster, which rejects the connection.

   3. Run the following commands to delete egress-by-egressgateway, and then access http://www.httpbin.org and https://www.httpbin.org again from the sleep-a service in the mytest namespace.

      kubectl -n istio-egress delete ASMEgressTrafficPolicy  egress-by-egressgateway
      kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  http://www.httpbin.org
      kubectl -n mytest exec deployment/sleep-a -- curl -s -o /dev/null -w "%{http_code}\n"  https://www.httpbin.org

      Requests to http://www.httpbin.org return 502, and requests to https://www.httpbin.org are rejected. After the ASMEgressTrafficPolicy configuration is deleted, the sleep-a service can no longer access http://www.httpbin.org or https://www.httpbin.org.

      These results confirm that the ASMEgressTrafficPolicy works as expected.

Deny POST requests from a specific namespace

kind: AuthorizationPolicy
apiVersion: security.istio.io/v1beta1
metadata:
  name: sleep-a-egress-www-httpbin-org
  namespace: istio-system
spec:
  action: DENY
  rules:
    - to:
        - operation:
            hosts:
            - www.httpbin.org
            - httpbin.org
            methods:
              - POST
      from:
        - source:
            namespaces: ["mytest"]
  selector:
    matchLabels:
      istio: egressgateway-a

After applying this configuration, POST requests from the sleep-a service to www.httpbin.org return RBAC: access. GET requests from the sleep-a service to www.httpbin.org are not affected.