Enable health check redirection for mesh applications - Alibaba Cloud Service Mesh

A sidecar intercepts the requests that a mesh application receives, so HTTP and TCP health checks for a sidecar-injected application in Alibaba Cloud Service Mesh (ASM) can behave unexpectedly, such as HTTP health checks that fail consistently. Enable health check redirection so that the health checks work as expected.

Background information

The following table describes the unexpected behavior of HTTP and TCP health checks for applications in the mesh. To restore the health checks, add an annotation to enable health check redirection.

Type

Description

HTTP health checks

In a Kubernetes cluster, the kubelet sends the health check requests of all pods. After mutual TLS (mTLS) mode is enabled, applications in the mesh must communicate over TLS. The kubelet is not part of the mesh and therefore does not have a certificate that ASM issues to applications. As a result, HTTP health check requests are rejected and the health checks fail consistently.

TCP health checks

To intercept requests, the sidecar listens on all ports of the pod of an application in the mesh. During a TCP health check, the kubelet determines the health status of the application by checking whether an application listens on the port configured for the application pod.

For this reason, the health check always succeeds as long as a sidecar is injected into the application and the sidecar is running, regardless of the actual state of the application. For example, if you configure an incorrect port for the application, the health check of the pod is supposed to always fail and keep the pod in the not-ready status, but the health check succeeds.

Note

If you do not enable mTLS mode in ASM, you can use HTTP health checks on application pods without configuring health check redirection.

By default, the mesh topology graph displays the health check request calls of application services. In many scenarios, these calls can confuse traffic statistics. Enable health check redirection for the applications in the mesh to remove these internal health check calls from the statistics.

How health check redirection works

Health check redirection is controlled by one annotation on the pod template of a workload. Add sidecar.istio.io/rewriteAppHTTPProbers: "true" under template.metadata.annotations, and then apply the workload. ASM rewrites the health checks configured for the application container as HTTP health checks on port 15020.

For applications in the mesh, 15020 is a special port that is used for mesh observability. Traffic sent to this port is not intercepted by the sidecar and therefore does not need to meet the requirements of the TLS mode. After health check redirection is enabled, the pilot-agent service that runs in the sidecar container listens on port 15020 and receives health checks from the kubelet. Based on the health check configuration in the ISTIO_KUBE_APP_PROBERS environment variable, the pilot-agent service forwards the health check requests to the application container. This way, HTTP health checks run as expected.

For TCP health checks, health check redirection in ASM performs the same processing and rewrites them as HTTP health checks on port 15020. Based on the TCP health check configuration in the ISTIO_KUBE_APP_PROBERS environment variable, the pilot-agent service probes the TCP health check port configured for the application container. If the actual TCP health check fails, the pilot-agent service returns the 500 status code to indicate that the health check failed.

Prerequisites

The application is added to ASM and a sidecar is injected into the application pods.
kubectl is connected to the cluster that runs the application. For instructions, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Enable HTTP health check redirection

The following example uses an Nginx application to enable HTTP health check redirection. After mTLS mode is enabled, the HTTP health check configured for the Nginx application fails consistently. Health check redirection is then enabled for the Nginx application. If the pod events contain no health check failure events and the pod is in the ready status, HTTP health check redirection is enabled for the application.

Step 1: Enable STRICT mTLS mode for a namespace

Configure mTLS mode on the PeerAuthentication page of the target ASM instance in the ASM console. The mTLS mode that you configure in this step applies to the namespace that you select.

Log on to the ASM console.
In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, find the ASM instance that you want to configure. Click the name of the ASM instance or click Manage in the Actions column.
On the details page of the ASM instance, choose Mesh Security Center > PeerAuthentication in the left-side navigation pane.
At the top of the PeerAuthentication page, select a namespace, and then click Configure Global mTLS Mode.
In the Configure Global mTLS Mode panel, set mTLS Mode (Namespace-wide) to STRICT - Strictly Enforce mTLS, and then click Create.

Step 2: Deploy the Nginx application

Deploy the Nginx application.

Create a file named http-liveness.yaml with the following content.

http-liveness.yaml: initial configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /index.html
            port: 80
            httpHeaders:
            - name: X-Custom-Header
              value: hello
          initialDelaySeconds: 5
          periodSeconds: 3

Under the readinessProbe parameter, the httpGet field defines an HTTP health check for the application.

Run the following command to deploy the Nginx application.
```
kubectl apply -f http-liveness.yaml
```

View the health check status of the application.
1. Run the following command to view the pod name of the Nginx application.
```
kubectl get pod | grep nginx
```
2. Run the following command to view the pod events.
```
kubectl describe pod <pod_name>
```
  Expected output:
```
Warning  Unhealthy  45s               kubelet            Readiness probe failed: Get "http://172.23.64.22:80/index.html": read tcp 172.23.64.1:54130->172.23.64.22:80: read: connection reset by peer
```
  The output shows that the HTTP health check of the pod failed, which keeps the pod in the not-ready status.

Step 3: Enable health check redirection for the Nginx application

Run the following command to edit http-liveness.yaml.

vim http-liveness.yaml

Add the following content under the template parameter:

annotations:
  sidecar.istio.io/rewriteAppHTTPProbers: "true"

The following code shows the http-liveness.yaml file after the annotation is added:

http-liveness.yaml: after the annotation is added

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /index.html
            port: 80
            httpHeaders:
            - name: X-Custom-Header
              value: hello
          initialDelaySeconds: 5
          periodSeconds: 3

Run the following command to deploy the Nginx application.
```
kubectl apply -f http-liveness.yaml
```

Step 4: Verify the health check result

View the health check status of the pod.
1. Run the following command to view the pod name of the Nginx application.
```
kubectl get pod | grep nginx
```
2. Run the following command to view the pod events.
```
kubectl describe pod <pod_name>
```
  The output does not include any health check failure events and the pod is in the ready status. The HTTP health check works as expected.

Run the following command to view the YAML file of the pod after health check redirection.

kubectl get pod <pod_name> -o yaml

Expected output

apiVersion: v1
kind: Pod
metadata:
  ...
  name: nginx-deployment-676f85f66b-cbzsx
  namespace: default
  ...
spec:
  containers:
    - args:
        - proxy
        - sidecar
        - '--domain'
        - $(POD_NAMESPACE).svc.cluster.local
        - '--proxyLogLevel=warning'
        - '--proxyComponentLogLevel=misc:error'
        - '--log_output_level=default:info'
        - '--concurrency'
        - '2'
      env:
        ...
        - name: ISTIO_KUBE_APP_PROBERS
          value: >-
            {"/app-health/nginx/readyz":{"httpGet":{"path":"/index.html","port":80,"scheme":"HTTP","httpHeaders":[{"name":"X-Custom-Header","value":"hello"}]},"timeoutSeconds":1}}
      ...
    - image: nginx
      imagePullPolicy: IfNotPresent
      name: nginx
      ports:
        - containerPort: 80
          protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          httpHeaders:
            - name: X-Custom-Header
              value: hello
          path: /app-health/nginx/readyz
          port: 15020
          scheme: HTTP
        initialDelaySeconds: 5
        periodSeconds: 3
        successThreshold: 1
        timeoutSeconds: 1

After health check redirection is enabled, the health check port changes from 80 to 15020, and the health check path changes from /index.html to /app-health/nginx/readyz. The ISTIO_KUBE_APP_PROBERS environment variable is also added to the sidecar container in the pod. The value of the variable is the JSON serialization of the health check configuration before the health check is rewritten.

Enable TCP health check redirection

The following example uses an Nginx application to enable TCP health check redirection. The procedure is the same as the procedure for HTTP health check redirection. Only the probe configuration and the expected results differ. An incorrect port is configured for the Nginx application, but the TCP health check still succeeds on the application pod, which does not meet expectations. After you enable health check redirection for the Nginx application, the TCP health check fails on the application pod, which meets expectations and indicates that TCP health check redirection is enabled for the application.

Step 1: Deploy the Nginx application

Deploy the Nginx application.
1. Create a file named tcp-liveness.yaml with the following content.
  
  The following content configures 2940 as the health check port, which is incorrect. The Nginx application does not listen on port 2940. Therefore, the health check is supposed to always fail after the application is deployed and keep the pod in the not-ready status.
  tcp-liveness.yaml: initial configuration
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        readinessProbe:
          tcpSocket:
            port: 2940
          initialDelaySeconds: 5
          periodSeconds: 3
```
  Under the readinessProbe parameter, the tcpSocket field defines a TCP health check for the application.
2. Run the following command to deploy the Nginx application.
```
kubectl apply -f tcp-liveness.yaml
```
View the health check status of the application.
1. Run the following command to view the pod name of the Nginx application.
```
kubectl get pod | grep nginx
```
2. Run the following command to view the pod events.
```
kubectl describe pod <pod_name>
```
  The output does not include any health check failure events and the pod is in the ready status, which does not meet expectations.

Step 2: Enable health check redirection for the Nginx application

Run the following command to edit the tcp-liveness.yaml file.

vim tcp-liveness.yaml

In the tcp-liveness.yaml file, add the following content under the template parameter:

annotations:
  sidecar.istio.io/rewriteAppHTTPProbers: "true"

The following code shows the tcp-liveness.yaml file after the annotation is added:

tcp-liveness.yaml: after the annotation is added

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        readinessProbe:
          tcpSocket:
            port: 2940
          initialDelaySeconds: 5
          periodSeconds: 3

Run the following command to deploy the Nginx application.
```
kubectl apply -f tcp-liveness.yaml
```

Step 3: Verify the health check result

View the health check status of the application.
1. Run the following command to view the pod name of the Nginx application.
```
kubectl get pod | grep nginx
```
2. Run the following command to view the pod events.
```
kubectl describe pod <pod_name>
```
  Expected output:
```
Warning  Unhealthy  45s               kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
```
  The output indicates that the health check failed, which meets expectations.

Run the following command to view the YAML content of the application pod after health check redirection.

kubectl get pod <pod_name> -o yaml

Expected output

apiVersion: v1
kind: Pod
metadata:
  ...
  name: nginx-deployment-746458cdc9-m9t9q
  namespace: default
  ...
spec:
  containers:
    - args:
        - proxy
        - sidecar
        - '--domain'
        - $(POD_NAMESPACE).svc.cluster.local
        - '--proxyLogLevel=warning'
        - '--proxyComponentLogLevel=misc:error'
        - '--log_output_level=default:info'
        - '--concurrency'
        - '2'
      env:
        ...
        - name: ISTIO_KUBE_APP_PROBERS
          value: >-
            {"/app-health/nginx/readyz":{"tcpSocket":{"port":2940},"timeoutSeconds":1}}
      ...
    - image: nginx
      imagePullPolicy: IfNotPresent
      name: nginx
      ports:
        - containerPort: 80
          protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /app-health/nginx/readyz
          port: 15020
          scheme: HTTP
        initialDelaySeconds: 5
        periodSeconds: 3
        successThreshold: 1
        timeoutSeconds: 1
      ...

After health check redirection is enabled, the original TCP health check is rewritten as an HTTP health check. The health check port changes from 2940 to 15020, and the path of the HTTP health check is set to /app-health/nginx/readyz. The ISTIO_KUBE_APP_PROBERS environment variable is also added to the sidecar container in the pod. The value of the variable is the JSON serialization of the TCP health check configuration before the health check is rewritten.