How to implement zero-downtime rolling deployments in Kubernetes - Container Service for Kubernetes

Configure readiness probes, readinessGates, preStop hooks, and Server Load Balancer (SLB) connection draining for zero-downtime rolling updates in ACK.

How it works

The Rolling Update strategy for Deployments replaces Pods one by one, ensuring continuous availability. The process has three phases:

Startup phase: A new Pod (v2) is created. Kubernetes waits for it to pass the readiness probe before routing any Service traffic to it.
Traffic shifting phase: With readinessGates enabled, the new Pod must pass its readiness check. Its IP is then registered with Service Endpoints and the SLB backend server group. The system then terminates the old Pod (v1) and removes its IP from Endpoints.

See How readinessGates works.
Graceful shutdown phase: Before deletion, the old Pod executes its preStop hook and uses terminationGracePeriodSeconds to finish processing connections, while the SLB drains in-flight requests. This ensures all requests complete for a zero-downtime update.

Prerequisites

Cluster version is 1.24 or later. Upgrade the cluster if needed.
The cloud-controller-manager component is v2.10.0 or later. See Cloud Controller Manager.

Deploy a sample application

This example deploys a stateless NGINX application.

Console

On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.

On the Deployments page, click Create from YAML. Paste the following YAML and click Create.

Sample application YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-demo
spec:
  replicas: 1                 # Set to 2 or more for production HA. Set to 1 for demonstration purposes.
  selector:
    matchLabels:
      app: nginx-demo
  # Rolling update strategy: ensures service availability during updates.
  # strategy:
    # type: RollingUpdate     # Default strategy for Deployments.
    # rollingUpdate:
      # maxUnavailable: "25%" # Default. Max 25% of Pods can be unavailable during the update.
      # maxSurge: "25%"       # Default. Max 25% extra Pods can be created above the desired replica count.
  template:
    metadata:
      labels:
        app: nginx-demo 
    spec:
      # Pod-level graceful shutdown limit. Must be greater than the sum of preStop execution and app cleanup time.
      terminationGracePeriodSeconds: 60 
      readinessGates:
      - conditionType: service.readiness.alibabacloud.com/nginx-demo-service # Set the Readiness Gate for the nginx-demo-service Service.
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 500m
        # --- Health check probes ---
        # startup probe: Ensures the application in the container has started.
        startupProbe:
          httpGet:
            path: / # Accessing the default NGINX root path indicates a successful startup.
            port: 80
          # Allow sufficient time for startup. Total timeout = failureThreshold * periodSeconds.
          # Here: 30 * 10 = 300 seconds.
          failureThreshold: 30
          periodSeconds: 10
        # readiness probe: Determines whether the container is ready to receive traffic.
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5  # Probing starts 5 seconds after the container starts.
          periodSeconds: 5        # Probe every 5 seconds.
          timeoutSeconds: 2       # Probe timeout duration.
          successThreshold: 1     # 1 success marks the Pod as ready.
          failureThreshold: 3     # 3 consecutive failures mark the Pod as not ready.
        # --- Pod graceful shutdown configuration ---
        lifecycle:
          preStop:
            exec:
              # For reliable graceful shutdown, define a custom hook that handles in-flight requests based on your application logic.
              # Using sleep alone is not recommended as it does not guarantee a clean exit.
              command: ["sh", "-c", "sleep 30 && /usr/sbin/nginx -s quit"]
---           
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-service
  annotations:
    # Timeout for connection draining. This value should align with the application's preStop logic. Range: 10-900.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30" 
    # Enable connection draining.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
spec:
  type: LoadBalancer
  selector:
    app: nginx-demo 
  ports:
    - protocol: TCP
      port: 80

In the pop-up window, locate the application, click View , and verify the Pod status is Running.

kubectl

Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Save the following YAML as nginx-demo.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-demo
spec:
  replicas: 1                 # Set to 2 or more for production HA. Set to 1 for demonstration purposes.
  selector:
    matchLabels:
      app: nginx-demo
  # Rolling update strategy: ensures service availability during updates.
  # strategy:
    # type: RollingUpdate     # Default strategy for Deployments.
    # rollingUpdate:
      # maxUnavailable: "25%" # Default. Max 25% of Pods can be unavailable during the update.
      # maxSurge: "25%"       # Default. Max 25% extra Pods can be created above the desired replica count.
  template:
    metadata:
      labels:
        app: nginx-demo 
    spec:
      # Pod-level graceful shutdown limit. Must be greater than the sum of preStop execution and app cleanup time.
      terminationGracePeriodSeconds: 60 
      readinessGates:
      - conditionType: service.readiness.alibabacloud.com/nginx-demo-service # Set the Readiness Gate for the nginx-demo-service Service.
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 500m
        # --- Health check probes ---
        # startup probe: Ensures the application in the container has started.
        startupProbe:
          httpGet:
            path: / # Accessing the default NGINX root path indicates a successful startup.
            port: 80
          # Allow sufficient time for startup. Total timeout = failureThreshold * periodSeconds.
          # Here: 30 * 10 = 300 seconds.
          failureThreshold: 30
          periodSeconds: 10
        # readiness probe: Determines whether the container is ready to receive traffic.
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5  # Probing starts 5 seconds after the container starts.
          periodSeconds: 5        # Probe every 5 seconds.
          timeoutSeconds: 2       # Probe timeout duration.
          successThreshold: 1     # 1 success marks the Pod as ready.
          failureThreshold: 3     # 3 consecutive failures mark the Pod as not ready.
        # --- Pod graceful shutdown configuration ---
        lifecycle:
          preStop:
            exec:
              # For reliable graceful shutdown, define a custom hook that handles in-flight requests based on your application logic.
              # Using sleep alone is not recommended as it does not guarantee a clean exit.
              command: ["sh", "-c", "sleep 30 && /usr/sbin/nginx -s quit"]
---           
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-service
  annotations:
    # Timeout for connection draining. This value should align with the application's preStop logic. Range: 10-900.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30" 
    # Enable connection draining.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
spec:
  type: LoadBalancer
  selector:
    app: nginx-demo 
  ports:
    - protocol: TCP
      port: 80

Deploy the NGINX application and create the Service.
```
kubectl apply -f nginx-demo.yaml
```

Verify the Pod is Running.

kubectl get pod | grep nginx-deployment-demo

Pod readiness checks
- startupProbe (startup probe): Checks whether slow-starting applications, such as Java applications, have launched. Until it succeeds, readiness and liveness probes are paused to prevent premature container restarts.
- readinessProbe (readiness probe): Determines whether a container can handle requests. On success, the Pod IP is added to Service Endpoints, enabling traffic.
- readinessGates: A Pod is considered ready only after both readinessProbe and readinessGates pass.
Graceful shutdown
- Application graceful shutdown
  - preStop: A hook that runs before container termination. Configure it to drain in-flight requests for a non-disruptive shutdown.
    
    Use a custom hook. Using sleep alone may not exit correctly.
  - terminationGracePeriodSeconds: Maximum time from Pod termination to SIGKILL. Default: 30 seconds. Must exceed the combined preStop hook and application cleanup time.
- SLB connection draining
  - service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain annotation: Enables SLB connection draining.
  - service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: Connection draining timeout in seconds. Set this close to the preStop hook processing time.
Rolling update strategy
- strategy: Default Deployment strategy is RollingUpdate. Gradually creates new Pods and removes old ones after they are ready, ensuring service availability.
- maxUnavailable: Maximum unavailable Pods during a rolling update. Default: 25%. Also accepts an absolute number.
- maxSurge: Maximum extra Pods above the desired replica count during a rolling update. Higher values speed up updates but consume more resources. Default: 25%. Also accepts an absolute number.

Verify the zero-downtime rolling deployment

Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Get the sample application URL.

export NGINX_ENDPOINT=$(kubectl get service nginx-demo-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}{":"}{.spec.ports[0].port}')
echo $NGINX_ENDPOINT

Install hey. Run a load test with 200 concurrent connections and 50,000 total requests. A single replica completes this in about one minute.
```
hey -c 200 -n 50000  -disable-keepalive http://$NGINX_ENDPOINT
```
While the test runs, open a new terminal and immediately restart the Deployment.
```
kubectl rollout restart deployment nginx-deployment-demo
```

Expected outputs:

Deployment scenario

Expected output

Without zero-downtime configuration

Sample YAML without zero-downtime configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-demo
spec:
  replicas: 1                 # Set to 2 or more for production HA. Set to 1 for demonstration purposes.
  selector:
    matchLabels:
      app: nginx-demo
  template:
    metadata:
      labels:
        app: nginx-demo 
    spec:
      containers:
      - name: nginx
        image: anolis-registry.cn-zhangjiakou.cr.aliyuncs.com/openanolis/nginx:1.14.1-8.6
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 500m
---           
apiVersion: v1
kind: Service
metadata:
  name: nginx-demo-service
spec:
  type: LoadBalancer
  selector:
    app: nginx-demo 
  ports:
    - protocol: TCP
      port: 80

Traffic loss occurs.

Status code distribution:
  [200]	49644 responses

Error distribution:
  [320]	Get "http://114.215.XXX.XXX": dial tcp 114.215.XXX.XXX:80: connect: connection refused
  [18]	Get "http://114.215.XXX.XXX": dial tcp 114.215.XXX.XXX:80: connect: no route to host
  [18]	Get "http://114.215.XXX.XXX": dial tcp 114.215.XXX.XXX:80: connect: operation timed out

With zero-downtime configuration

No traffic loss.

Status code distribution:
  [200]	50000 responses

FAQ

Pod status: Running but not ready

Cause: Usually caused by a failed startup or readiness probe.

Solution:

Readiness probe configuration: On the Edit page of the target Workloads, verify the health check path (for example, /healthz) and port match the application. For slow-starting applications, increase the Unhealthy Threshold to avoid premature failures.

Temporarily disable the Readiness check, log on to the Pod terminal, and use a command such as curl to verify the health check responds correctly.
Troubleshoot application issues: Check the Pod's Events and Logs. Select Show the log of the last container exit.

Container Service for Kubernetes:Achieve zero-downtime rolling updates with graceful shutdown