Obtain client source IP in Alibaba Cloud Service Mesh - Alibaba Cloud Service Mesh

This document describes how to configure Service Mesh to preserve the client source IP when your services are accessed.

Prerequisites

You have an ASM Enterprise or Ultimate Edition instance, version 1.15 or later. For more information, see Create an ASM instance and Upgrade an ASM instance.
An ACK managed cluster is created. For more information, see Create an ACK managed cluster.
The cluster is added to the ASM instance.
An ingress gateway is deployed. For more information, see Create an ingress gateway.
You are connected to the cluster by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Background information

The client source IP is used in many scenarios, including:

Application access control: For example, many applications enforce additional authentication when a user logs on from a different region, which requires obtaining the client's original IP.
Simple session affinity: You can perform source IP-based load balancing to forward requests from the same client to the same service instance.
Access logging and monitoring: Access logs and monitoring metrics that contain the real source address help developers analyze traffic and gather statistics.

Cloud load balancers like Server Load Balancer (SLB) can pass the client source IP to backend services. Istio should also provide this capability. However, when you use Istio, a sidecar proxy is injected into each pod. This proxy intercepts all inbound traffic, forwarding it to your application over a local connection (127.0.0.1). As a result, your application cannot see the real client source IP.

Deploy sample applications

Deploy the sleep application.

Create a file named sleep.yaml with the following content.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Run the following command to deploy the sleep application.
```
kubectl -n default apply -f  sleep.yaml
```

Deploy the httpbin application.

Create a file named httpbin.yaml with the following content.

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/citizenstig/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8000

Run the following command to deploy the httpbin application.
```
kubectl -n default apply -f  httpbin.yaml
```

Scenario 1: East-west traffic

Step 1: Verify the default client source IP behavior

In Istio, the sidecar proxy for each service intercepts all east-west traffic. This proxy then forwards requests to the application container, so the source address the application sees is the proxy's loopback address, 127.0.0.6.

Run the following command to check the pod status.

kubectl -n default get pods -o wide

Expected output:

NAME                            READY   STATUS        RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
httpbin-c85bdb469-4ll2m         2/2     Running       0          3m22s   172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>
sleep-8f764df66-q7dr2           2/2     Running       0          3m9s    172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>

The output shows that the address of the sleep application is 172.17.X.XXX.

Run the following command to send a request from the sleep container.
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected output:
```
{
  "origin": "127.0.0.6"
}
```
The output shows that the source address of the request received by the httpbin application is the Envoy proxy's address 127.0.0.6, not the address of the sleep application.
Confirm the source IP address is 127.0.0.6 by inspecting the socket information.
1. Log on to the httpbin container and run the following command to install netstat.
```
apt update & apt install net-tools
```
2. Exit the httpbin container and run the following command to view information about port 8000.
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 8000
```
  Expected output:
```
tcp        0      0 172.17.X.XXX:8000         127.0.0.6:42691         TIME_WAIT   -
```
  The output shows that the source IP address is 127.0.0.6.

View the content of the access log in the httpbin pod.

The following is a sample formatted log entry:

{
  "trace_id":null,
  "bytes_received":0,
  "upstream_host":"172.17.X.XXX:8000",
  "authority":"httpbin:8000",
  "downstream_remote_address":"172.17.X.XXX:56160",
  "upstream_service_time":"1",
  "upstream_transport_failure_reason":null,
  "istio_policy_status":null,
  "path":"/ip",
  "bytes_sent":28,
  "request_id":"4501a50a-dab0-44c9-b52c-2a4f425a****",
  "protocol":"HTTP/1.1",
  "method":"GET",
  "duration":1,
  "start_time":"2022-11-22T16:09:30.394Z",
  "user_agent":"curl/7.86.0-DEV",
  "upstream_local_address":"127.0.0.6:42169",
  "response_flags":"-",
  "route_name":"default",
  "response_code":200,
  "upstream_cluster":"inbound|80||",
  "x_forwarded_for":null,
  "downstream_local_address":"172.17.X.XXX:8000",
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local"
}

The log shows the following information:

"downstream_remote_address":"172.17.X.XXX:56160": The address of the sleep application.
"downstream_local_address":"172.17.X.XXX:8000": The destination address that the sleep application accesses.
"upstream_local_address":"127.0.0.6:42169": The local address that the httpbin Envoy proxy uses to connect to the httpbin application. At this point, the source IP seen by the application is 127.0.0.6.
"upstream_host":"172.17.X.XXX:8000": The destination address that the httpbin Envoy proxy accesses.

Step 2: Configure source IP preservation

Method 1: Use TPROXY mode

Configure the httpbin deployment to use the TPROXY transparent interception mode.

Run the following command to modify the deployment of the httpbin application.

kubectl patch deployment -n default httpbin -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/interceptionMode":"TPROXY"}}}}}'

Run the following command to send a request from the sleep container.
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected output:
```
{
  "origin": "172.17.X.XXX"
}
```
The output shows that the httpbin application can obtain the real IP address of the sleep application.
Run the following command to view information about port 8000.
Note
After the pod restarts, you must reinstall netstat.
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 8000
```
Expected output:
```
tcp        0      0 172.17.X.XXX:8000         172.17.X.XXX:36728      ESTABLISHED -
```
The output shows that the source IP address is 172.17.X.XXX.

View the content of the access log in the httpbin pod.

The following is a sample formatted log entry:

{
  "route_name":"default",
  "bytes_received":0,
  "trace_id":null,
  "request_id":"1ccabe60-63cf-469b-8565-99cac546****",
  "upstream_cluster":"inbound|80||",
  "response_flags":"-",
  "protocol":"HTTP/1.1",
  "upstream_transport_failure_reason":null,
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
  "response_code":200,
  "user_agent":"curl/7.86.0-DEV",
  "start_time":"2022-11-22T16:03:32.803Z",
  "path":"/ip",
  "authority":"httpbin:8000",
  "bytes_sent":31,
  "downstream_remote_address":"172.17.X.XXX:39058",
  "upstream_service_time":"1",
  "method":"GET",
  "downstream_local_address":"172.17.X.XXX:8000",
  "duration":1,
  "upstream_host":"172.17.X.XXX:8000",
  "istio_policy_status":null,
  "upstream_local_address":"172.17.X.XXX:46129",
  "x_forwarded_for":null
}

The log shows the following information:

"downstream_remote_address":"172.17.X.XXX:39058": The address of the sleep application.
"downstream_local_address":"172.17.X.XXX:8000": The destination address that the sleep application accesses.
"upstream_local_address":"172.17.X.XXX:46129": The local address that the httpbin Envoy proxy uses to connect to the httpbin application, which is the IP address of the sleep application.
"upstream_host":"172.17.X.XXX:8000": The destination address that the httpbin Envoy proxy accesses.

Method 2: Use the XFF header

The following configuration makes the server-side sidecar proxy add an X-Forwarded-For (XFF) header to inbound requests. The proxy sets this header's value to the client's real IP address before forwarding the request to the application. This method has no OS limitations but requires your application to read the client's source IP from the X-Forwarded-For (XFF) header.

Apply the following EnvoyFilter to the ASM instance by using an EnvoyFilter template. For more information, see Create an Envoy filter by using an Envoy filter template.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: enable-xff-for-sidecar-inbound
  namespace: istio-system  # Change this to the namespace where your gateway is located.
  labels:
    asm-system: "true"
    provider: "asm"
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      proxy:
        proxyVersion: "^1.*"
      context: SIDECAR_INBOUND 
      listener:
        name: "virtualInbound"
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          use_remote_address: true

Run the following command to send a request from the sleep container.

kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip

Expected output:
```
{
  "origin": "172.17.X.XXX"
}
```
The output shows that httpbin receives the sleep application's real IP address. The httpbin application can do this because it is designed to read the source IP from the X-Forwarded-For (XFF) header, which the sidecar proxy now adds. This method will not work if your application cannot read the XFF header.

Scenario 2: North-south traffic

For north-south traffic, requests flow from a client, through a load balancer and the Istio ingress gateway, to the backend service. This extra hop through the ingress gateway makes preserving the client source IP more complex. The following sections describe how to configure and verify source IP preservation for HTTP and HTTPS requests.

HTTP requests

Without source IP preservation

Create a file named http-demo.yaml with the following content to access the httpbin application over HTTP.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-httpprotocol
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
    - httpbin-gw-httpprotocol
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the gateway and VirtualService.
```
kubectl -n default apply -f  http-demo.yaml
```

Run the following command to access the httpbin application through the ingress gateway.

export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$GATEWAY_URL:80/ip

Expected output:

{
  "origin": "10.0.0.93"
}

The output shows that the returned IP address is the address of a Kubernetes node.

View the access log of the ingress gateway.

The following is a sample log entry:

{
  "upstream_service_time":"1",
  "response_code":200,
  "protocol":"HTTP/1.1",
  "bytes_sent":28,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "start_time":"2022-11-23T03:29:20.017Z",
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "trace_id":null,
  "route_name":null,
  "request_id":"292903be-a889-4d5d-83a0-ab1f5d1a****",
  "method":"GET",
  "upstream_host":"172.17.X.XXX:8000",
  "duration":1,
  "path":"/ip",
  "downstream_local_address":"172.17.X.XXX:80",
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_remote_address":"10.0.0.93:5899",
  "upstream_local_address":"172.17.X.XXX:54322",
  "requested_server_name":null,
  "x_forwarded_for":"10.0.0.93",
  "response_flags":"-",
  "bytes_received":0
}

The log shows the following information:

"downstream_remote_address":"10.0.0.93:5899": Not the client's real source IP.
"downstream_local_address":"172.17.X.XXX:80": The address of the ingress gateway pod.
"upstream_local_address":"172.17.X.XXX:54322": The IP address of the ingress gateway pod is preserved, but the port number is changed.
"upstream_host":"172.17.X.XXX:8000": The address of the httpbin pod.

With source IP preservation

Set the external traffic policy to Local. (You can skip this step for clusters that use the Terway network mode.)
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
3. On the Ingress Gateway page, click View YAML to the right of the target gateway.
4. In the Edit dialog box, find the spec section, set the externalTrafficPolicy field to Local, and click OK.
```
spec:
  affinity: {}
  autoCreateGatewayYaml: false
  clusterIds:
    - cf0243f2c3009406xxx
  compression: {}
  cpu: {}
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  gatewayType: ingress
```
Run the following command to access the httpbin application through the ingress gateway.
```
curl http://$GATEWAY_URL:80/ip
```
Expected output:
```
{
  "origin": "120.244.xxx.xxx"
}
```
The output shows that the returned IP address is the actual client source IP.

View the access log of the ingress gateway.

The following is a sample log entry:

{
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "path":"/ip",
  "x_forwarded_for":"120.244.XXX.XXX",
  "route_name":null,
  "method":"GET",
  "duration":2,
  "downstream_remote_address":"120.244.XXX.XXX:28504",
  "bytes_received":0,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "bytes_sent":34,
  "protocol":"HTTP/1.1",
  "response_flags":"-",
  "upstream_local_address":"172.17.X.XXX:57498",
  "upstream_service_time":"2",
  "request_id":"9c0295d4-e77f-4a3a-b292-e5c58d92****",
  "start_time":"2022-11-23T03:24:04.413Z",
  "response_code":200,
  "trace_id":null,
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_local_address":"172.17.X.XXX:80",
  "upstream_host":"172.17.X.XXX:80",
  "requested_server_name":null
}

The log shows the following information:

"downstream_remote_address":"120.244.XXX.XXX:28504": The client source address, as expected.
"downstream_local_address":"172.17.X.XXX:80": The address of the ingress gateway pod.
"upstream_local_address":"172.17.X.XXX:57498": The address of the ingress gateway pod is preserved, but the port number is changed.
"upstream_host":"172.17.X.XXX:80": The address of the httpbin pod.

HTTPS requests

Since the previous section detailed HTTP requests, this section focuses only on the configuration and verification for HTTPS requests.

Configure source IP preservation.

Create a file named https-demo.yaml with the following content to access the httpbin application over HTTPS.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-https
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: myexample-credential
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-https
  namespace: default
spec:
  gateways:
    - httpbin-gw-https
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the gateway and VirtualService.
```
kubectl -n default apply -f  https-demo.yaml
```

Run the following command to access the httpbin application through the ingress gateway.

export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k https://$GATEWAY_URL:443/ip

Expected output:

{
  "origin": "120.244.XXX.XXX"
}

The output shows that the returned IP address is the actual client source IP.