Graceful shutdown keeps the shutdown and restart of an online application transparent to clients: normal business requests are not affected from the moment the application stops until the application restarts and resumes service. Configure graceful shutdown when you deploy, stop, roll back, scale in, or reset an application so that the application closes properly. This topic describes how graceful shutdown works, how to enable and observe it, and the usage notes that apply.
Why you need graceful shutdown
In a microservice architecture, if you shut down a provider instance directly during periods of high concurrency, consumer instances cannot detect the status change of the provider instance in time. Consumers keep forwarding requests to an instance that is already offline, which causes request errors and traffic loss.
How it works
In the following description, provider A is the application that shuts down and consumer B is the application that calls it. When provider A shuts down, it notifies the Microservices Registry, but consumer B might not detect the change in real time and keeps calling provider A after it goes offline. To prevent this, provider A enters a wait period as soon as it receives the shutdown command and adds a special marker to the responses of all requests that it receives during the wait period. When consumer B receives such a response and identifies the marker, it pulls the service instances from the Microservices Registry once and detects the latest status of provider A in real time. Provider A also waits for a period of time before it shuts down, so that all in-flight requests received before the shutdown are processed instead of being cut off when provider A stops.
For any application that is connected to MSE, graceful shutdown is enabled by default, but proactive notification must be configured manually. MSE Graceful Start and Shutdown also provides observability that helps you determine whether the application shut down gracefully.
When a microservice application in a Kubernetes environment is connected to MSE Microservices Governance, graceful shutdown operations are injected into the Pod as a lifecycle.preStop hook and run before the Pod stops. Therefore, we recommend that you do not configure a custom preStop hook. If your preStop hook only deregisters the application from the Microservices Registry, you can remove it and use the preStop hook that graceful shutdown injects automatically.
If your business requires graceful termination operations and you have already configured a custom preStop hook for the application container, graceful shutdown no longer injects its preStop hook into your application container. Instead, graceful shutdown injects a sidecar container named gracefulshutdown and injects the graceful shutdown preStop hook into that container. Because a sidecar container shares the network namespace with the application container by default, running the preStop hook of the sidecar container before the Pod stops also achieves graceful shutdown for the application container.
To make sure that the preStop hook of the sidecar container can finish, include a sleep of at least 30 seconds in the custom preStop hook of the application container so that in-flight requests are processed. Set the Pod-level terminationGracePeriodSeconds parameter accordingly, as described in Usage notes.
The sidecar container consumes minimal resources. The current settings are CPU: 50m, Memory: 50Mi.
In Kubernetes, graceful shutdown is implemented based on the
preStophook. Therefore, it currently supports graceful shutdown only for normal Pod termination, such as scale-in, restart, and rolling upgrade. It does not support graceful shutdown for abnormal termination, such as an out-of-memory (OOM) kill.
Enable graceful shutdown
Before you enable graceful shutdown, review Prerequisites and Usage notes. Enable graceful shutdown based on the environment in which your application is deployed:
If your application is deployed in a Container Service for Kubernetes (ACK) environment, no additional operation is required to enable graceful shutdown after the application is connected to MSE Microservices Governance. By default, graceful shutdown is enabled automatically. However, if the application container already has a custom
preStophook, or if the Pod still uses the defaultterminationGracePeriodSecondsvalue of 30, you must also complete the configurations described in How it works and Usage notes.If your application is deployed in an Elastic Compute Service (ECS) environment, add the following content to the shutdown script of the application and run it early in the script. The request to
/offlinestarts the graceful shutdown process, andsleep 30leaves time for in-flight requests to be processed before the script continues.curl http://127.0.0.1:54199/offline 2>/tmp/null; sleep 30;
To confirm that graceful shutdown takes effect, see Observe graceful shutdown.
Observe graceful shutdown
After graceful shutdown takes effect for your application, you can observe on the Application Governance page that traffic to a terminating instance is drained within a very short time and that its QPS drops to 0 quickly.
Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose . On the page that appears, click the resource card of the application that you want to manage.
On the application details page, click Traffic management in the left-side navigation pane, and click the Graceful Start/Shutdown tab.
On the Start and Shutdown Overview tab, view the events that occurred during recent starts and shutdowns of the application. In the list of started and shut down instances on the left, find the terminating instance that you want to view, and then click it.
The details on the right show that the instance ran the graceful shutdown process, that traffic was drained quickly after the process finished, and that no further traffic entered the instance before it stopped.
If the QPS does not drop to 0 quickly after a successful graceful shutdown event is generated, first check whether traffic comes from sources other than microservice calls, such as local calls.
The "application stopped" event is reported only by agent version 4.2.0 or later. If you do not see this event, consider upgrading the agent.
Proactive notification
What is proactive notification
Proactive notification is an advanced feature of graceful shutdown and is disabled by default. In most scenarios, the shutdown solution that MSE Graceful Start and Shutdown provides already resolves the problem. Enable proactive notification if the application to be shut down uses the Spring Cloud framework and consumer calls fail when the application shuts down. After you enable proactive notification, a provider instance proactively notifies service consumers during the shutdown phase, and the notified consumers stop sending requests to that provider instance.
Prerequisites
Usage notes
MSE Microservices Governance does not yet support graceful shutdown for the following applications:
Applications that are not developed in Java.
Applications that are not developed based on WebFlux or Spring MVC.
Downstream provider applications whose consumers are not microservice applications.
Applications whose consumer application or provider application is not connected to MSE Microservices Governance. Both the consumer application and the provider application must be connected to MSE Microservices Governance to achieve graceful shutdown.
The graceful shutdown process waits for a period of time to make sure that all in-flight requests to the terminating instance are processed. Therefore, the value of the terminationGracePeriodSeconds parameter in your Kubernetes Pod specification (default value: 30) must be greater than 30. We recommend that you set it to 90. If you use the default value 30, the application shutdownhook may not finish running, and some resources cannot be released properly when the application stops.
Enable proactive notification
Log on to the MSE console, and select a region in the top navigation bar.
In the left-side navigation pane, choose . On the page that appears, click the resource card of the application that you want to manage.
On the application details page, click Traffic management in the left-side navigation pane, and click the Graceful Start/Shutdown tab.
In the Settings section, click Modify, click the Graceful Shutdown section, turn on Proactive Notification, and then click OK.
To confirm that proactive notification takes effect, observe the next shutdown of the application as described in Observe graceful shutdown.