Preserve the source IP address of a client when the client accesses services in ASM - Alibaba Cloud Service Mesh

This topic describes how to preserve the source IP address of a client when the client accesses services in Service Mesh (ASM).

Prerequisites

An ASM instance of Enterprise Edition or Ultimate Edition is created, and the version of the ASM instance is 1.15 or later. For more information, see Create an ASM instance and Update an ASM instance.
An ACK managed cluster is created. For more information, see Create an ACK managed cluster.
The cluster is added to the ASM instance. For more information, see Add a cluster to an ASM instance.
An ingress gateway is deployed. For more information, see Create an ingress gateway.
The Container Service for Kubernetes (ACK) cluster is connected by using kubectl. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Background information

Source IP addresses are widely used. Typical usage scenarios include:

Access control for applications: Many applications enforce additional identity verification by checking the source IP address when they detect that a user is trying to log on from a different location.
Session persistence: You can configure a load balancer to direct all the requests from a client to the same server instance based on the source IP address of the client.
Access logs and monitoring statistics: Access logs and monitoring statistics that contain real source IP addresses help developers analyze the information.

A load balancer on the cloud can transfer the source IP address of a client to the backend service. Istio is supposed to allow applications to obtain the original source IP addresses. However, when Envoy proxies are injected as sidecars in Istio, the proxies redirect all inbound traffic. Envoy sends the traffic to the applications that are bound to the localhost 127.0.0.1, so the applications cannot obtain the original source IP addresses.

Deploy sample applications

Deploy a sleep application.

Create a sleep.yaml file that contains the following content:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Run the following command to deploy the sleep application:
```
kubectl -n default apply -f  sleep.yaml
```

Deploy an HTTPBin application.

Create an httpbin.yaml file that contains the following content:

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/citizenstig/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8000

Run the following command to deploy the HTTPBin application:
```
kubectl -n default apply -f  httpbin.yaml
```

Scenario 1: East-west traffic

TPROXY mode is not enabled

Envoy proxies are injected as sidecars in Istio. These proxies intercept all inbound and outbound traffic for services and forward the traffic to requested applications. Therefore, the IP addresses of the requests received by applications are the endpoint that is used to access Envoy 127.0.0.6.

Run the following command to query the pod status:

kubectl -n default get pods -o wide

Expected output:

NAME                            READY   STATUS        RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
httpbin-c85bdb469-4ll2m         2/2     Running       0          3m22s   172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>
sleep-8f764df66-q7dr2           2/2     Running       0          3m9s    172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>

The output shows that the IP address of the sleep application is 172.17.X.XXX.

Run the following command to initiate a request from the container where the sleep application resides:
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected output:
```
{
  "origin": "127.0.0.6"
}
```
The output shows that the IP address of the request received by the HTTPBin application is the endpoint that is used to access Envoy 127.0.0.6 instead of the IP address of the sleep application.
Check whether the source IP address is 127.0.0.6 from the socket.
1. Log on to the container where the HTTPBin application resides and run the following command to install netstat:
```
apt update & apt install net-tools
```
2. Log off from the container. Then, run the following command to connect to the HTTPBin container and view information about the connection that uses port 80:
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80
```
  Expected output:
```
tcp        0      0 172.17.X.XXX:80         127.0.0.6:42691         TIME_WAIT   -
```
  The output shows that the source IP address is 127.0.0.6.

View proxy logs of the pod where the HTTPBin application resides.

The following example shows the formatted logs:

{
  "trace_id":null,
  "bytes_received":0,
  "upstream_host":"172.17.X.XXX:80",
  "authority":"httpbin:8000",
  "downstream_remote_address":"172.17.X.XXX:56160",
  "upstream_service_time":"1",
  "upstream_transport_failure_reason":null,
  "istio_policy_status":null,
  "path":"/ip",
  "bytes_sent":28,
  "request_id":"4501a50a-dab0-44c9-b52c-2a4f425a****",
  "protocol":"HTTP/1.1",
  "method":"GET",
  "duration":1,
  "start_time":"2022-11-22T16:09:30.394Z",
  "user_agent":"curl/7.86.0-DEV",
  "upstream_local_address":"127.0.0.6:42169",
  "response_flags":"-",
  "route_name":"default",
  "response_code":200,
  "upstream_cluster":"inbound|80||",
  "x_forwarded_for":null,
  "downstream_local_address":"172.17.X.XXX:80",
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local"
}

The logs indicate the following information:

"downstream_remote_address":"172.17.X.XXX:56160": the IP address of the sleep application.
"downstream_local_address":"172.17.X.XXX:80": the IP address that the sleep application requests to access.
"upstream_local_address":"127.0.0.6:42169": the localhost IP address that the Envoy proxy for the HTTPBin application uses to connect to HTTPBin. In this case, the source IP address obtained is 127.0.0.6.
"upstream_host":"172.17.X.XXX:80": the IP address that the Envoy proxy for the HTTPBin application requests to access.

Enable TPROXY mode to preserve source IP addresses

Modify the deployment of the HTTPBin application to intercept inbound traffic by using TPROXY.

Run the following command to modify the deployment of the HTTPBin application:

kubectl patch deployment -n default httpbin -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/interceptionMode":"TPROXY"}}}}}'

Run the following command to initiate a request from the container where the sleep application resides:
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected output:
```
{
  "origin": "172.17.X.XXX"
}
```
The output shows that the HTTPBin application can get the real IP address of the sleep application.
Run the following command to connect to the HTTPBin container and view information about the connection that uses port 80.
Note
After the pod is restarted, you must reinstall netstat.
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80
```
Expected output:
```
tcp        0      0 172.17.X.XXX:80         172.17.X.XXX:36728      ESTABLISHED -
```
The output shows that the source IP address is 172.17.X.XXX.

View proxy logs of the pod where the HTTPBin application resides.

The following example shows the formatted logs:

{
  "route_name":"default",
  "bytes_received":0,
  "trace_id":null,
  "request_id":"1ccabe60-63cf-469b-8565-99cac546****",
  "upstream_cluster":"inbound|80||",
  "response_flags":"-",
  "protocol":"HTTP/1.1",
  "upstream_transport_failure_reason":null,
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
  "response_code":200,
  "user_agent":"curl/7.86.0-DEV",
  "start_time":"2022-11-22T16:03:32.803Z",
  "path":"/ip",
  "authority":"httpbin:8000",
  "bytes_sent":31,
  "downstream_remote_address":"172.17.X.XXX:39058",
  "upstream_service_time":"1",
  "method":"GET",
  "downstream_local_address":"172.17.X.XXX:80",
  "duration":1,
  "upstream_host":"172.17.X.XXX:80",
  "istio_policy_status":null,
  "upstream_local_address":"172.17.X.XXX:46129",
  "x_forwarded_for":null
}

The logs indicate the following information:

"downstream_remote_address":"172.17.X.XXX:39058": the IP address of the sleep application.
"downstream_local_address":"172.17.X.XXX:80": the IP address that the sleep application requests to access.
"upstream_local_address":"172.17.X.XXX:46129": the localhost IP address that the Envoy proxy for the HTTPBin application uses to connect to HTTPBin, that is, the IP address of the sleep application.
"upstream_host":"172.17.X.XXX:80": the IP address that the Envoy proxy for the HTTPBin application requests to access.

Scenario 2: North-south traffic

For north-south traffic, a client request is first sent to the load balancer, forwarded to the ingress gateway, and then forwarded to the backend service. Because the ingress gateway is added to the process, it becomes more complicated to obtain the source IP address of the client. The following sections describe how to configure and verify the preservation of source IP addresses for HTTP and HTTPS requests.

HTTP requests

Source IP addresses are not preserved

Create a http-demo.yaml file that contains the following content to access the HTTPBin application over HTTP:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-httpprotocol
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
    - httpbin-gw-httpprotocol
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the Istio gateway and virtual service:
```
kubectl -n default apply -f  http-demo.yaml
```
Run the following command to access the HTTPBin application by using the ingress gateway:
```
export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$GATEWAY_URL:80/ip
```
Expected output:
```
{
  "origin": "10.0.0.93"
}
```
The output shows that the returned IP address is the node address of the Kubernetes cluster.

View the access logs of the ingress gateway.

Sample logs:

{
  "upstream_service_time":"1",
  "response_code":200,
  "protocol":"HTTP/1.1",
  "bytes_sent":28,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "start_time":"2022-11-23T03:29:20.017Z",
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "trace_id":null,
  "route_name":null,
  "request_id":"292903be-a889-4d5d-83a0-ab1f5d1a****",
  "method":"GET",
  "upstream_host":"172.17.X.XXX:80",
  "duration":1,
  "path":"/ip",
  "downstream_local_address":"172.17.X.XXX:80",
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_remote_address":"10.0.0.93:5899",
  "upstream_local_address":"172.17.X.XXX:54322",
  "requested_server_name":null,
  "x_forwarded_for":"10.0.0.93",
  "response_flags":"-",
  "bytes_received":0
}

The logs indicate the following information:

"downstream_remote_address":"10.0.0.93:5899": is not the source IP address of the client.
"downstream_local_address":"172.17.X.XXX:80": the IP address of the pod where the ingress gateway resides.
"upstream_local_address":"172.17.X.XXX:54322": retains the IP address of the pod where the ingress gateway resides but changes the port number.
"upstream_host":"172.17.X.XXX:80": the IP address of the pod where the HTTPBin application resides.

Preserve source IP addresses

Configure an external traffic routing policy. If your cluster uses the Terway network plug-in, skip this step.
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
3. On the Ingress Gateway page, find the ingress gateway that you want to use and click YAML.
4. In the Edit dialog box, set the externalTrafficPolicy field in the spec section to Local and then click OK.
Run the following command to access the HTTPBin application by using the ingress gateway:
```
curl http://$GATEWAY_URL:80/ip
```
Expected output:
```
{
  "origin": "120.244.xxx.xxx"
}
```
The output shows that the returned IP address is the source IP address of the client.

View the access logs of the ingress gateway.

Sample logs:

{
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "path":"/ip",
  "x_forwarded_for":"120.244.XXX.XXX",
  "route_name":null,
  "method":"GET",
  "duration":2,
  "downstream_remote_address":"120.244.XXX.XXX:28504",
  "bytes_received":0,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "bytes_sent":34,
  "protocol":"HTTP/1.1",
  "response_flags":"-",
  "upstream_local_address":"172.17.X.XXX:57498",
  "upstream_service_time":"2",
  "request_id":"9c0295d4-e77f-4a3a-b292-e5c58d92****",
  "start_time":"2022-11-23T03:24:04.413Z",
  "response_code":200,
  "trace_id":null,
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_local_address":"172.17.X.XXX:80",
  "upstream_host":"172.17.X.XXX:80",
  "requested_server_name":null
}

The logs indicate the following information:

"downstream_remote_address":"120.244.XXX.XXX:28504": the source IP address of the client, which meets expectations.
"downstream_local_address":"172.17.X.XXX:80": the IP address of the pod where the ingress gateway resides.
"upstream_local_address":"172.17.X.XXX:57498": retains the IP address of the pod where the ingress gateway resides and changes the port number.
"upstream_host":"172.17.X.XXX:80": the IP address of the pod where the HTTPBin application resides.

HTTPS requests

The preceding section compares the outcomes before and after the preservation of source IP addresses is configured. This section only describes how to configure and verify the preservation of source IP addresses for HTTPS requests.

Configure the preservation of source IP addresses.

Create an https-demo file that contains the following content to access the HTTPBin application over HTTPS:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-https
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: myexample-credential
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-https
  namespace: default
spec:
  gateways:
    - httpbin-gw-https
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the Istio gateway and virtual service:
```
kubectl -n default apply -f  https-demo.yaml
```

Run the following command to access the HTTPBin application by using the ingress gateway:

export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k https://$GATEWAY_URL:443/ip

Expected output:

{
  "origin": "120.244.XXX.XXX"
}

The output shows that the returned IP address is the source IP address of the client.