Why health checks fail or become invalid after Sidecar injection - Alibaba Cloud Service Mesh

When mutual TLS (mTLS) is enabled in Service Mesh (ASM), the sidecar proxy intercepts all inbound traffic to the pod, including health check requests from kubelet. Because kubelet does not have a mesh-issued TLS certificate, health checks fail or return misleading results.

Issue description

After sidecar injection with mTLS enabled, health checks stop working. For example, with a TCP health check on port 8087, no health check events appear on the Events tab of the pod details page in the Container Service for Kubernetes (ACK) console.

Each Kubernetes probe type reacts differently when the sidecar proxy intercepts traffic:

Probe type	Behavior with sidecar and mTLS	Impact
HTTP	Fails. kubelet cannot present a valid TLS certificate to the sidecar.	Pod is marked unhealthy and may restart.
TCP	Always succeeds. The sidecar listens on the port, so the TCP connection succeeds regardless of the application state.	Health check becomes meaningless. The application could be down, but the probe still passes.
gRPC	Fails. Same TLS certificate mismatch as HTTP probes.	Pod is marked unhealthy and may restart.
Command	Unaffected. Command probes run inside the container and do not pass through the sidecar.	No action needed.

Causes

When mTLS is enabled, the sidecar proxy requires all inbound connections to present a valid TLS certificate issued by the mesh certificate authority. kubelet sends health check requests directly to the pod and does not have this certificate. As a result:

HTTP and gRPC probes fail the TLS handshake and the sidecar rejects them.
TCP probes succeed at the connection level because the sidecar itself listens on the port, masking actual application health.

This is expected behavior in Envoy-based service meshes, not a defect. The sidecar enforces mTLS on all traffic, and kubelet operates outside the mesh trust domain.

Solution

Configure the health check port to bypass the sidecar proxy. Traffic on bypassed ports goes directly to the application container without mTLS enforcement.

Bypassed ports are no longer protected by mTLS. Only bypass ports used for health checks, not ports that carry application traffic.

Configure health check traffic to bypass the sidecar proxy

Log on to the ASM console.
In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, find the target ASM instance and click its name, or click Manage in the Actions column.
In the left-side navigation pane of the details page, choose Data Plane Component Management > Sidecar Proxy Setting.
On the Namespace tab, select the namespace where the affected pod runs.

Click the Enable/disable Sidecar Proxy by Ports or Addresses tab and configure the following parameters: Replace 8087 with the actual port used by your health check.

Parameter	Description	Example
Ports on Which Inbound Traffic Not Redirected to Sidecar Proxy	Ports for inbound traffic that bypass the sidecar proxy.	`8087`
Ports on Which Outbound Traffic Not Redirected to Sidecar Proxy	Ports for outbound traffic that bypass the sidecar proxy.	`8087`

Click Update Settings.
Restart the affected pods so that the updated sidecar configuration takes effect.

Verify health check results

After the pods restart, confirm that health checks work correctly.

Option A: Check pod events in the ACK console

Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the target cluster and click its name, or click Details in the Actions column.
In the left-side navigation pane, choose Workloads > Pods.
Click the target pod name, or click Details to open the pod details page.
Click the Events tab and verify that health check events for the configured port appear.

Option B: Check pod events with kubectl

Run the following command:

kubectl describe pod <pod-name> -n <namespace>

In the Events section of the output, look for Liveness or Readiness probe entries. Successful health checks show no probe failure warnings.

Placeholder	Description	Example
`<pod-name>`	Name of the affected pod	`my-app-6d8f7b9c4-xk2mn`
`<namespace>`	Namespace where the pod runs	`default`