Use readiness gates to seamlessly launch pods that are associated with ALB Ingresses during rolling updates - Container Compute Service

During a rolling update, you need to make sure that the updated pods are ready before they are launched to receive traffic. You can configure readiness gates to ensure the readiness of pods before they are launched. The Application Load Balancer (ALB) Ingress controller allows you to enable readiness gates to continuously monitor the status of pods in an Alibaba Cloud Container Compute Service (ACS) cluster. After the status of the pods changes to Ready, the pods are launched and added to a backend server group. Then, traffic is forwarded to the pods. This topic describes how to seamlessly launch pods that are associated with an ALB Ingress during a rolling update by comparing the results before and after readiness gates are used.

Background information

Liveness, readiness, and startup probes

ACS uses liveness probes, readiness probes, and startup probes to check the health status, readiness, and startup conditions of pods in order to improve the availability of applications. You can configure these types of probes by using HTTP requests, TCP sockets, or commands. You can specify the probing interval, timeout period, healthy threshold, and unhealthy threshold. For more information, see Configure Liveness, Readiness and Startup Probes.

Readiness gates

If readiness probes are used, the value of the Ready field of a pod is determined by the kubelet based on the status of the containers in the pod. However, for complex applications, you may need to control the readiness of Services in containers in a fine-grained manner. To do this, you need to control the value of the Ready field of pods.

Readiness gates allow you to set one or more custom readiness conditions to check whether a pod is ready. These custom conditions are set on an external controller. Kubernetes considers that a pod is ready only if all readinessGates conditions are True. For more information, see Pod Readiness Gate.

You can enable readiness gates on the ALB Ingress controller. After you set the readinessGates condition to target-health.alb.k8s.alibabacloud in a Deployment, the ACS cluster will continuously monitor the pods. The ACS cluster considers that a pod is ready and starts to forward traffic to the pod only after the ALB Ingress controller adds the pod to a backend server group. This way, backend pods can be seamlessly launched during rolling updates.

Prerequisites

The ALB Ingress controller is installed in the ACS cluster. For more information, see Manage the ALB Ingress controller.
An AlbConfig and an IngressClass are created. For more information, see Create an AlbConfig and Create an IngressClass.
Connect to your ACS cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster and Use kubectl on Cloud Shell to manage ACS clusters.

Procedure

Step 1: Deploy an application named tea

Create a file named tea-service.yaml based on the following content. In this example, a Deployment named tea and a Service named tea-svc are created.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tea
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginxdemos:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: tea
  type: ClusterIP

Run the following command to deploy the Deployment and the Service:
```
kubectl apply -f tea-service.yaml
```

Run the following command to check the status of the application:

kubectl get pods -o wide

Expected results:

NAME                   READY   STATUS    RESTARTS   AGE    IP                NODE  NOMINATED NODE   READINESS GATES
tea-5cb56xxxxx-xxxxx   1/1     Running   0          1m4s   192.168.xxx.xxx   xxx   <none>           <none>

If the READINESS GATES field of pods displays <none>, readinessGates are not configured.

Create a file named tea-ingress.yaml based on the following content.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tea-ingress
spec:
  ingressClassName: alb
  rules:
   - host: www.example.com
     http:
      paths:
      - path: /tea
        pathType: Prefix
        backend:
          service:
            name: tea-svc
            port:
              number: 80

Run the following command to create the Ingress:
```
kubectl apply -f tea-ingress.yaml
```

Run the following command to query the information of the ALB Ingress:

kubectl get ingress

Expected results:

NAME          CLASS   HOSTS             ADDRESS                                              PORTS   AGE
tea-ingress   alb     www.example.com   alb-qu066wzmi5fbixxxxx.cn-xxxxxxx.alb.aliyuncs.com   80      6m47s

Note

The domain name www.example.com is used as an example. For the actual domain name, refer to Configure domain name resolution.

Step 2: Verify that the application tea is interrupted during rolling updates

Execute a test script.

Create a script named test.sh and copy the following content to the script. The script is used only to test the availability of the application by continuously sending requests to the application. You can check the returned HTTP status codes.

#!/bin/bash
HOST="www.example.com"
DNS="alb-qu066wzmi5fbixxxxx.cn-xxxxxxx.alb.aliyuncs.com"   # Specify the value of the ADDRESS parameter of the ALB Ingress. 
while true; do
  RESPONSE=$(curl -H Host:$HOST  -s -o /dev/null -w "%{http_code}" -m 1  http://$DNS/tea)
  TIMESTAMP=$(date +%Y-%m-%d_%H:%M:%S)
  echo "$TIMESTAMP - $RESPONSE" 
done

Run the following command to execute the test.sh script:
```
bash test.sh
```

Run the following command to start a rolling update for the application and redeploy the application:
```
kubectl rollout restart deploy tea 
```
Verify the rolling update.
The execution result shows a small number of 502 (Bad Gateway) status codes. This means that the gateway service (ALB Ingress) provided by the application tea is temporarily interrupted.

Step 3: Deploy an application named tea-readiness and configure readiness gates

Create a file named tea-readiness-service.yaml based on the following content. In this example, a Deployment named tea-readiness and a Service named tea-readiness-svc are created. The .spec.readinessGates.conditionType parameter of pods is set to target-health.alb.k8s.alibabacloud to enable readiness gates for the ALB Ingress controller.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea-readiness
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tea-readiness
  template:
    metadata:
      labels:
        app: tea-readiness
    spec:
      containers:
      - name: tea-readiness
        image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginxdemos:latest
        ports:
        - containerPort: 80
#Configure readiness gates.
      readinessGates:
        - conditionType: target-health.alb.k8s.alibabacloud
---
apiVersion: v1
kind: Service
metadata:
  name: tea-readiness-svc
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: tea-readiness
  type: ClusterIP

Run the following command to deploy the Deployment and the Service:
```
kubectl apply -f tea-readiness-service.yaml
```

Run the following command to check the status of the application:

kubectl get pods -o wide

Expected results:

NAME                             READY   STATUS    RESTARTS   AGE    IP                NODE   NOMINATED NODE   READINESS GATES
tea-5cb56xxxxx-xxxxx             1/1     Running   0          11m    192.168.xxx.xxx   xxx    <none>           <none>
tea-readiness-5cb56xxxxx-xxxxx   1/1     Running   0          4m7s   192.168.xxx.xxx   xxx    <none>           0/1

No Ingress is configured. Therefore, the READINESS GATES field of the application tea-readiness displays 0/1, which indicates that the readinessGates configuration has not taken effect.

Create a file named tea-readiness-ingress.yaml based on the following content.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tea-readiness-ingress
spec:
  ingressClassName: alb
  rules:
   - host: www.example.com
     http:
      paths:
      - path: /tea-readiness
        pathType: Prefix
        backend:
          service:
            name: tea-readiness-svc
            port:
              number: 80

Run the following command to create the Ingress:
```
kubectl apply -f tea-readiness-ingress.yaml
```

Run the following command to query the information of the ALB Ingress:

kubectl get ingress

Expected results:

NAME                    CLASS   HOSTS             ADDRESS                                              PORTS   AGE
tea-ingress             alb     www.example.com   alb-qu066wzmi5fbi5lg85.cn-beijing.alb.aliyuncs.com   80      12m
tea-readiness-ingress   alb     www.example.com   alb-qu066wzmi5fbi5lg85.cn-beijing.alb.aliyuncs.com   80      65s

Step 4: Verify that the pods of the application tea-readiness can be seamlessly launched during rolling updates

Run the following command to check the status of the application: The READINESS GATES field of the application tea-readiness displays 1/1, which indicates that the readinessGates configuration is in effect.

kubectl get pods -o wide

Expected results:

NAME                             READY   STATUS    RESTARTS   AGE    IP                NODE   NOMINATED NODE   READINESS GATES
tea-5cb56xxxxx-xxxxx             1/1     Running   0          13m    192.168.xxx.xxx   xxx    <none>           <none>
tea-readiness-5cb56xxxxx-xxxxx   1/1     Running   0          6m2s   192.168.xxx.xxx   xxx    <none>           1/1

Execute a test script.

Create a script named test-readiness.sh and copy the following content to the script. The script is used only to test the availability of the application by continuously sending requests to the application. You can check the returned HTTP status codes.

#!/bin/bash
HOST="www.example.com"
DNS="alb-qu066wzmi5fbixxxxx.cn-xxxxxxx.alb.aliyuncs.com"   # Specify the value of the ADDRESS parameter of the ALB Ingress. 
while true; do
  RESPONSE=$(curl -H Host:$HOST  -s -o /dev/null -w "%{http_code}" -m 1  http://$DNS/tea-readiness)
  TIMESTAMP=$(date +%Y-%m-%d_%H:%M:%S)
  echo "$TIMESTAMP - $RESPONSE" 
done

Run the following command to execute the test-readiness.sh script:
```
bash test-readiness.sh
```

Run the following command to start a rolling update for the application and redeploy the application:
```
kubectl rollout restart deploy tea-readiness
```
Verify the rolling update.
The execution result indicates that only the status code 200 is returned. The application is not interrupted during rolling updates.