When a pod with an injected sidecar proxy is terminated, long-running requests to and from the pod can be dropped or fail. Learn the cause of these issues and how to resolve them.
Symptoms
After a sidecar proxy is injected into a pod, the following two issues occur when the pod is terminated:
-
Some long-running requests sent to the pod are dropped.
-
Requests that the pod sends to other services might fail.
Cause
After a sidecar proxy is injected into a pod, istio-proxy proxies the traffic of the application pod. When the pod starts to terminate, the corresponding Service no longer forwards traffic to the pod.
In Istio, istio-proxy is forcibly stopped 5 seconds after it receives a termination signal by default. Within these 5 seconds, istio-proxy stops accepting new inbound connections (inbound traffic) but continues to process existing inbound connections. Outbound connections (outbound traffic) are not affected during this period and can still be initiated. When the 5 seconds elapse, istio-proxy stops. If calls to the service being terminated take a long time to process, the existing inbound and outbound connections are terminated before the requests on them are fully processed.
Solutions
Both of the following solutions give existing inbound and outbound connections more time to finish before istio-proxy is stopped, which happens 5 seconds after the termination signal by default. If you can estimate the maximum wait duration of your requests, extend the termination drain duration as described in Solution 1. If you cannot estimate that duration, configure a preStop script as described in Solution 2.
Solution 1: Extend the sidecar proxy termination drain duration
Extend the termination drain duration of the sidecar proxy so that inbound and outbound connections can be fully processed within the duration.
-
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
-
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Configure the agent parameters of the injected Sidecar.
-
On the Configure the agent parameters of the injected Sidecar page, click the Namespace tab.
-
Select a Namespace, click Lifecycle Management, select Sidecar Proxy Drain Duration at Pod Termination, enter a duration, and click Update Settings.
Solution 2: Configure a preStop script for the sidecar proxy lifecycle
If you cannot estimate the maximum wait duration of requests, configure a preStop script for the lifecycle of the sidecar proxy. The preStop script checks whether request connections still exist. Once no request connections remain, the sidecar proxy waits for the default duration of 5 seconds and then exits.
-
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
-
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Configure the agent parameters of the injected Sidecar.
-
On the Configure the agent parameters of the injected Sidecar page, click the Namespace tab.
-
Select a Namespace, click Lifecycle Management, select Lifecycle of Sidecar Proxy, enter the following content in the text box, and click Update Settings.
{
"postStart": {
"exec": {
"command": [
"pilot-agent",
"wait"
]
}
},
"preStop": {
"exec": {
"command": [
"/bin/sh",
"-c",
"while [ $(netstat -plunt | grep tcp | grep -v envoy | wc -l | xargs) -ne 0 ]; do sleep 1; done"
]
}
}
}