Container Compute Service:Use preStop hooks to enable seamless pod termination during rolling updates of backend pods of ALB Ingresses

Prerequisites

• An Alibaba Cloud Container Compute Service (ACS) cluster is created.

• The ALB Ingress controller is installed in the cluster.

• The kubeconfig file of the cluster is obtained and kubectl is used to connect to the cluster.

• An AlbConfig and an IngressClass are created. For more information, see Getting started with ALB Ingress.

How it works

Pod lifecycle

A pod may contain the following types of containers:

• Init container: Init containers run before main containers to complete specific initialization operations.

• Main container: Main containers run the processes of applications.

The following figure shows the lifecycle of a pod.

1. Run init containers: Init containers run before main containers to complete specific initialization operations.

2. Run main containers:

   1. The main container invokes the postStart hook function after the container is started.

   2. The system performs a liveness probe and a readiness probe on the main container.

   3. The main container invokes the preStop hook function before the container is terminated.

Pod termination procedure and traffic routing rule update procedure

When you delete a pod, the system performs the pod termination procedure and the traffic routing rule update procedure at the same time. The following section describes the procedures:

Pod terminationprocedure

1. kube-apiserver receives the deletion request and marks the pod as Terminating.

2. If a preStop hook is configured for the pod, the system executes the preStop hook.

3. The cluster sends a SIGTERM signal to the container.

4. The system waits until the container is terminated or until the graceful shutdown process of the pod times out.

   Note

   The default grace period (terminationGracePeriodSeconds) of a pod is 30 seconds.

5. If the pod is not terminated within the grace period, the system sends a SIGKILL signal to the container.

6. The pod is deleted.

Traffic routing rule update procedure

1. kube-apiserver receives the deletion request and marks the pod as Terminating.

2. The endpoint controller deletes the IP address of the pod from the endpoint.

3. The ALB Ingress controller reconciles the backend servers and removes the Service endpoint from the backend server group. This way, traffic is no longer routed to the deleted pod.

Use a preStop hook to seamlessly terminate a pod

During a rolling update, if the old pod is not seamlessly terminated, the following status codes may be returned when you send requests to the service. This compromises the service availability.

To prevent the preceding issues, perform the following operations:

Add a preStop hook to the Deployment to specify a sleep duration that starts after kube-apiserver receives the deletion request. The sleep duration ensures a sufficient period of time for the container to update traffic routing rules and the ALB Ingress controller to reconcile the backend servers and remove the old pod from the backend server group before the pod receives the SIGTERM signal. This way, the pod can be seamlessly terminated to prevent service interruptions during rolling updates and restarts.

The default grace period of a pod is 30 seconds. This indicates that the pod can continue to run for 30 seconds after it receives the SIGTERM signal. If the sum of the program shutdown duration and the amount of time required to complete the operations defined by the preStop hook exceeds 30 seconds, the graceful shutdown of the pod times out. After the grace period ends, the kubelet waits for 2 seconds and sends a SIGKILL signal to the pod to forcefully terminate the pod.

Step 1: Configure a preStop hook for a pod

1. Create a file named tea-service.yaml and copy the following content to the file.

   The file includes the preStop hook that is added to ensure seamless pod termination.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: tea
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: tea
     template:
       metadata:
         labels:
           app: tea
       spec:
         containers:
         - name: tea
           image: registry.cn-hangzhou.aliyuncs.com/acs-sample/nginxdemos:latest
           ports:
           - containerPort: 80
           lifecycle:  # Specify a preStop hook function that requires kube-apiserver to wait for 10 seconds before sending the SIGTERM signal to the pod. 
             preStop:  # The preStop hook configurations. 
               exec:  # Perform the operations defined by the preStop hook by running commands. 
                 command:  # The command to run. 
                 - /bin/sh
                 - -c
                 - "sleep 10"  # Sleep for 10 seconds. 
         terminationGracePeriodSeconds: 45  # Specify the grace period of the pod. 
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: tea-svc
   spec:
     ports:
     - port: 80
       targetPort: 80
       protocol: TCP
     selector:
       app: tea
     type: ClusterIP

2. Run the following command to deploy the Deployment and the Service:

   kubectl apply -f tea-service.yaml

   Expected output:

   deployment "tea" created
   service "tea-svc" created

3. Create a file named tea-ingress.yaml and copy the following content to the file. The file is used to create an Ingress.

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
   metadata:
     name: tea-ingress
   spec:
     ingressClassName: alb
     rules:
      - host: example.com
        http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: tea-svc
               port:
                 number: 80

4. Run the following command to create the Ingress:

   kubectl apply -f tea-ingress.yaml

5. Run the following command to query the ADDRESS parameter of the ALB Ingress:

   kubectl get ingress

   Expected output:

   NAME                    CLASS   HOSTS                     ADDRESS                                              PORTS   AGE
   tea-ingress             alb     example.com          alb-110zvs5nhsvfv*****.cn-chengdu.alb.aliyuncs.com   80      7m5s

Step 2: Test whether the preStop hook takes effect

1. Execute a test script.

   1. Create a script named test.sh and copy the following content to the script. The script is used to test the service availability by sending requests to the NGINX service on a per-second basis and recording the returned status codes.

      #!/bin/bash
      HOST="example.com"
      DNS="alb-110zvs5nhsvfv*****.cn-chengdu.alb.aliyuncs.com"   # Set the value to the value of the ADDRESS parameter of the ALB Ingress. 
      while true; do
        RESPONSE=$(curl -H Host:$HOST  -s -o /dev/null -w "%{http_code}" -m 1  http://$DNS/)
        TIMESTAMP=$(date +%Y-%m-%d_%H:%M:%S)
        echo "$TIMESTAMP - $RESPONSE" 
        sleep 1
      done

   2. Run the following command to execute the test.sh script:

      bash test.sh

2. Run the following command to start a rolling update for the application and redeploy the application:

   kubectl rollout restart deploy tea 

   Note

   Open the CLI that runs the script and the CLI that verifies the rolling upgrade to see whether the service can be accessed during the application update.

3. Verify the update.

   1. The number of replicated pods before the rolling update is three.

   2. During the redeployment, two old pods still run when new pods are not ready to process requests.

   3. After the new pods are ready to process requests and added to the backend server group of the ALB instance, the old pods are terminated.

      

   4. After all old pods complete running the preStop hook function or the graceful shutdown of the pods times out, the kubectl sends a signal to the old pods to terminate them. After the pods are terminated, the rolling update is completed.

      

4. View the execution result of the test script. If 200 is returned for each request, no service interruption occurs during the rolling update.