Alibaba Cloud Service Mesh:Why do time-consuming requests get lost or fail after a sidecar proxy is injected into a pod?

Problem description

Causes

After a sidecar proxy is injected into a pod, the traffic of the pod is controlled by the sidecar proxy. After the pod stops, the corresponding Service no longer routes the traffic to the pod.

By default, Istio forcibly stops the sidecar proxy 5 seconds after Istio receives the signal that the pod is stopped. After the sidecar proxy is stopped, no inbound traffic is received, but the established inbound connections can still be processed. Outbound connections are not affected and can be initiated as expected. If the requests to the pod of the Service that is stopped are time-consuming, the existing inbound and outbound connections are terminated regardless of whether the connections are processed.

Solutions

Solution 1: Modify the termination drain duration for sidecar proxies

You can extend the termination drain duration for sidecar proxies so that inbound and outbound connections can be processed within this period.

1. Log on to the ASM console.
2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
4. On the details page of the ASM instance, choose Sidecar Management(Data Plane) > Sidecar Proxy Setting in the left-side navigation pane.
5. On the Sidecar Proxy Setting page, click Namespace.
6. Select a namespace from the Namespace drop-down list. Click Lifecycle Management, enter a proper value in the Sidecar proxy termination duration time field, and then click Update Settings.

Solution 2: Configure the lifecycle of sidecar proxies

If you cannot estimate the maximum amount of time to wait for a request, we recommend that you set the preStop parameter to configure the lifecycle of sidecar proxies. The preStop parameter is used to check whether a request exists. If no requests exist, the sidecar proxy stops after the default amount of time, which is 5 seconds, elapses.

1. Log on to the ASM console.
2. In the left-side navigation pane, choose Service Mesh > Mesh Management.
3. On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
4. On the details page of the ASM instance, choose Sidecar Management(Data Plane) > Sidecar Proxy Setting in the left-side navigation pane.
5. On the Sidecar Proxy Setting page, click Namespace.
6. Select a namespace from the Namespace drop-down list. Click Lifecycle Management, enter the following content in the Sidecar proxy lifecycle code editor, and then click Update Settings.

   {
     "postStart": {
       "exec": {
         "command": [
           "pilot-agent",
           "wait"
         ]
       }
     },
     "preStop": {
       "exec": {
         "command": [
           "/bin/sh",
           "-c",
           "while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do sleep 1; done"
         ]
       }
     }
   }