How does the server application obtain the client source IP in a service mesh environment - Alibaba Cloud Service Mesh

This topic describes how server applications can obtain and preserve the client source IP address when accessing services in a Service Mesh environment.

Prerequisites

You need an ASM Enterprise Edition or Ultimate Edition instance with version 1.15 or later. For more information, see Create an ASM instance and Upgrade an ASM instance.
An ACK managed cluster is created. For more information, see Create an ACK managed cluster.
The cluster is added to the ASM instance.
An ingress gateway is deployed. For more information, see Create an ingress gateway.
The cluster is connected through kubectl. For more information, see Obtain the KubeConfig file of a cluster and connect to the cluster by using kubectl.

Background information

The originating IP address is widely used in many scenarios, including the following typical use cases:

Application access control: For example, many applications enforce additional authentication when they detect that users log on from different regions. This can be achieved by obtaining the originating IP address.
Simple session persistence: You can perform load balancing based on the source IP address according to the client address. This way, requests from the same client are forwarded to the same service instance.
Access logs and monitoring statistics: Access logs and metrics that contain the originating IP address help developers analyze statistics.

Cloud-based Server Load Balancer also supports passing the client source IP to backend services. Istio should provide the capability for applications to obtain the originating IP address. However, when using Istio, after the Sidecar proxy is injected into a pod, all inbound traffic is redirected from Envoy. Currently, Envoy sends traffic to applications that are bound to the local address (127.0.0.1), so applications cannot see the actual originating IP address.

Deploy sample applications

Deploy the sleep application.

Use the following content to create sleep.yaml.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Run the following command to deploy the sleep application.
```
kubectl -n default apply -f  sleep.yaml
```

Deploy the httpbin application.

Use the following content to create httpbin.yaml.

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/citizenstig/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8000

Run the following command to deploy the httpbin application.
```
kubectl -n default apply -f  httpbin.yaml
```

Scenario 1: East-west traffic

Step 1: Verify the client source IP address seen by the server in the default configuration

In Istio, when east-west services are accessed, all traffic in and out of services is intercepted and proxied by Envoy due to the injection of Sidecar. Then, Envoy forwards the requests to the application. Therefore, the source address of the requests received by the application is the access address of Envoy 127.0.0.6.

Run the following command to check the pod status.

kubectl -n default get pods -o wide

Expected result:

NAME                            READY   STATUS        RESTARTS   AGE     IP             NODE                     NOMINATED NODE   READINESS GATES
httpbin-c85bdb469-4ll2m         2/2     Running       0          3m22s   172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>
sleep-8f764df66-q7dr2           2/2     Running       0          3m9s    172.17.X.XXX   cn-hongkong.10.0.0.XX    <none>           <none>

From the expected result, you can see that the address of the sleep application is 172.17.X.XXX.

Run the following command to send a request from the sleep container.
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected result:
```
{
  "origin": "127.0.0.6"
}
```
From the expected result, you can see that the source address of the request received by the httpbin application is the access address of Envoy 127.0.0.6, not the address of the sleep application.
Confirm whether the source IP address is 127.0.0.6 from the socket information.
1. Log on to the httpbin container and run the following command to install netstat.
```
apt update & apt install net-tools
```
2. Exit the httpbin container and run the following command to view the running information based on port 80.
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80
```
  Expected result:
```
tcp        0      0 172.17.X.XXX:80         127.0.0.6:42691         TIME_WAIT   -
```
  From the expected result, you can see that the source IP address is 127.0.0.6.

View the proxy log content in the httpbin pod.

The formatted log example is as follows:

{
  "trace_id":null,
  "bytes_received":0,
  "upstream_host":"172.17.X.XXX:80",
  "authority":"httpbin:8000",
  "downstream_remote_address":"172.17.X.XXX:56160",
  "upstream_service_time":"1",
  "upstream_transport_failure_reason":null,
  "istio_policy_status":null,
  "path":"/ip",
  "bytes_sent":28,
  "request_id":"4501a50a-dab0-44c9-b52c-2a4f425a****",
  "protocol":"HTTP/1.1",
  "method":"GET",
  "duration":1,
  "start_time":"2022-11-22T16:09:30.394Z",
  "user_agent":"curl/7.86.0-DEV",
  "upstream_local_address":"127.0.0.6:42169",
  "response_flags":"-",
  "route_name":"default",
  "response_code":200,
  "upstream_cluster":"inbound|80||",
  "x_forwarded_for":null,
  "downstream_local_address":"172.17.X.XXX:80",
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local"
}

From the log, you can obtain the following information:

"downstream_remote_address":"172.17.X.XXX:56160": the address of sleep.
"downstream_local_address":"172.17.X.XXX:80": the destination address accessed by sleep.
"upstream_local_address":"127.0.0.6:42169": the local address of the httpbin Envoy connecting to httpbin (the source IP address obtained at this time is 127.0.0.6).
"upstream_host":"172.17.X.XXX:80": the destination address accessed by the httpbin Envoy.

Step 2: Configure the service mesh to allow the server to obtain the correct client source IP address

Method 1: Preserve the client source IP address through TPROXY transparent interception mode (applicable to non-CentOS operating systems)

Modify the deployment of the httpbin application to use TPROXY as the inbound traffic interception mode.

Run the following command to modify the deployment of the httpbin application.

kubectl patch deployment -n default httpbin -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/interceptionMode":"TPROXY"}}}}}'

Run the following command to send a request from the sleep container.
```
kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip
```
Expected result:
```
{
  "origin": "172.17.X.XXX"
}
```
From the expected result, you can see that httpbin can obtain the actual IP address of sleep.
Run the following command to view the running information based on port 80.
Note
After the pod is restarted, you need to reinstall netstat.
```
kubectl -n default exec -it deploy/httpbin -c httpbin -- netstat -ntp | grep 80
```
Expected result:
```
tcp        0      0 172.17.X.XXX:80         172.17.X.XXX:36728      ESTABLISHED -
```
From the expected result, you can see that the source IP address is 172.17.X.XXX.

View the proxy log content in the httpbin pod.

The formatted log example is as follows:

{
  "route_name":"default",
  "bytes_received":0,
  "trace_id":null,
  "request_id":"1ccabe60-63cf-469b-8565-99cac546****",
  "upstream_cluster":"inbound|80||",
  "response_flags":"-",
  "protocol":"HTTP/1.1",
  "upstream_transport_failure_reason":null,
  "requested_server_name":"outbound_.8000_._.httpbin.default.svc.cluster.local",
  "response_code":200,
  "user_agent":"curl/7.86.0-DEV",
  "start_time":"2022-11-22T16:03:32.803Z",
  "path":"/ip",
  "authority":"httpbin:8000",
  "bytes_sent":31,
  "downstream_remote_address":"172.17.X.XXX:39058",
  "upstream_service_time":"1",
  "method":"GET",
  "downstream_local_address":"172.17.X.XXX:80",
  "duration":1,
  "upstream_host":"172.17.X.XXX:80",
  "istio_policy_status":null,
  "upstream_local_address":"172.17.X.XXX:46129",
  "x_forwarded_for":null
}

From the log, you can obtain the following information:

"downstream_remote_address":"172.17.X.XXX:39058": the address of sleep.
"downstream_local_address":"172.17.X.XXX:80": the destination address accessed by sleep.
"upstream_local_address":"172.17.X.XXX:46129": the local address of the httpbin Envoy connecting to httpbin (the IP address of sleep).
"upstream_host":"172.17.X.XXX:80": the destination address accessed by the httpbin Envoy.

Method 2: Obtain the client source IP address through the XFF request header

You can use the following configuration to make the mesh proxy Sidecar of the server application add the X-Forwarded-For request header to the request before forwarding the inbound request to the application. The value of the request header is set to the actual client IP address. This method has no restrictions on the node operating system, but requires the application to be able to identify and obtain the client source IP address carried by the XFF request header.

Refer to Create an Envoy filter by using an Envoy filter template to apply the following EnvoyFilter to the ASM instance through the EnvoyFilter template.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: enable-xff-for-sidecar-inbound
  namespace: istio-system  # Change this namespace to the namespace where the gateway is located
  labels:
    asm-system: "true"
    provider: "asm"
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      proxy:
        proxyVersion: "^1.*"
      context: SIDECAR_INBOUND 
      listener:
        name: "virtualInbound"
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          use_remote_address: true

Run the following command to send a request from the sleep container.

kubectl -n default exec -it deploy/sleep -c sleep -- curl http://httpbin:8000/ip

Expected result:
```
{
  "origin": "172.17.X.XXX"
}
```
From the expected result, you can see that httpbin can obtain the actual IP address of sleep. Note that the client source IP address is passed to the httpbin application by Sidecar through the X-Forwarded-For request header. The httpbin application supports obtaining the client source IP address from the X-Forwarded-For request header, so it can correctly read the client source IP address. If the application does not have the ability to obtain the source IP address from the X-Forwarded-For request header, this method is not applicable.

Scenario 2: North-south traffic

For north-south traffic, the client first requests Server Load Balancer, which then forwards the request to the Istio ingressgateway, and then to the backend service. Due to the additional ingressgateway in the middle, obtaining the client source IP address becomes more complex. The following sections describe how to configure and verify the preservation of the source IP address for HTTP and HTTPS protocol requests.

HTTP protocol requests

Without Source IP Preservation

Use the following content to create http-demo.yaml to access httpbin using the HTTP protocol.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-httpprotocol
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
    - httpbin-gw-httpprotocol
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the gateway and virtual service.
```
kubectl -n default apply -f  http-demo.yaml
```
Run the following command to access httpbin through the ingressgateway.
```
export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl http://$GATEWAY_URL:80/ip
```
Expected result:
```
{
  "origin": "10.0.0.93"
}
```
From the expected result, you can see that the returned IP address is the node address of the Kubernetes cluster.

View the access log of the ingressgateway.

Sample log:

{
  "upstream_service_time":"1",
  "response_code":200,
  "protocol":"HTTP/1.1",
  "bytes_sent":28,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "start_time":"2022-11-23T03:29:20.017Z",
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "trace_id":null,
  "route_name":null,
  "request_id":"292903be-a889-4d5d-83a0-ab1f5d1a****",
  "method":"GET",
  "upstream_host":"172.17.X.XXX:80",
  "duration":1,
  "path":"/ip",
  "downstream_local_address":"172.17.X.XXX:80",
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_remote_address":"10.0.0.93:5899",
  "upstream_local_address":"172.17.X.XXX:54322",
  "requested_server_name":null,
  "x_forwarded_for":"10.0.0.93",
  "response_flags":"-",
  "bytes_received":0
}

From the log, you can obtain the following information:

"downstream_remote_address":"10.0.0.93:5899": not the actual client source address.
"downstream_local_address":"172.17.X.XXX:80": the address of the ingressgateway pod.
"upstream_local_address":"172.17.X.XXX:54322": the address of the ingressgateway pod is preserved, but the port value changes.
"upstream_host":"172.17.X.XXX:80": the address of the httpbin pod.

With Source IP Preservation

Set the external traffic policy to Local. (Skip this step for clusters with the Terway network mode.)
1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
3. On the Ingress Gateway page, click View YAML on the right side of the target gateway.
4. In the Edit dialog box, under the spec field, set the externalTrafficPolicy field to Local, and then click OK.
Run the following command to access httpbin through the ingressgateway.
```
curl http://$GATEWAY_URL:80/ip
```
Expected result:
```
{
  "origin": "120.244.xxx.xxx"
}
```
From the expected result, you can see that the returned IP address is the actual client source IP address.

View the access log of the ingressgateway.

Sample log:

{
  "istio_policy_status":null,
  "upstream_transport_failure_reason":null,
  "path":"/ip",
  "x_forwarded_for":"120.244.XXX.XXX",
  "route_name":null,
  "method":"GET",
  "duration":2,
  "downstream_remote_address":"120.244.XXX.XXX:28504",
  "bytes_received":0,
  "upstream_cluster":"outbound|8000||httpbin.default.svc.cluster.local",
  "bytes_sent":34,
  "protocol":"HTTP/1.1",
  "response_flags":"-",
  "upstream_local_address":"172.17.X.XXX:57498",
  "upstream_service_time":"2",
  "request_id":"9c0295d4-e77f-4a3a-b292-e5c58d92****",
  "start_time":"2022-11-23T03:24:04.413Z",
  "response_code":200,
  "trace_id":null,
  "authority":"47.242.XXX.XX",
  "user_agent":"curl/7.79.1",
  "downstream_local_address":"172.17.X.XXX:80",
  "upstream_host":"172.17.X.XXX:80",
  "requested_server_name":null
}

From the log, you can obtain the following information:

"downstream_remote_address":"120.244.XXX.XXX:28504": the client source address, as expected.
"downstream_local_address":"172.17.X.XXX:80": the address of the ingressgateway pod.
"upstream_local_address":"172.17.X.XXX:57498": the address of the ingressgateway pod is preserved, but the port value changes.
"upstream_host":"172.17.X.XXX:80": the address of the httpbin pod.

HTTPS protocol requests

The previous section has explained in detail the comparison before and after setting source IP preservation for HTTP protocol requests. Therefore, this section only describes how to configure and verify source IP preservation for HTTPS protocol requests.

Set source IP preservation.

Use the following content to create https-demo to access httpbin using the HTTPS protocol.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gw-https
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: myexample-credential
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-https
  namespace: default
spec:
  gateways:
    - httpbin-gw-https
  hosts:
    - '*'
  http:
    - route:
        - destination:
            host: httpbin
            port:
              number: 8000

Run the following command to deploy the gateway and virtual service.
```
kubectl -n default apply -f  https-demo.yaml
```

Run the following command to access httpbin through the ingressgateway.

export GATEWAY_URL=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -k https://$GATEWAY_URL:443/ip

Expected result:

{
  "origin": "120.244.XXX.XXX"
}

From the expected result, you can see that the returned IP address is the actual client source IP address.