Online applications must be developed in a way to ensure normal service requests even during the period from when applications are stopped for service upgrade and deployment to when services are restarted and recovered. That is, the entire process must be imperceptible to clients. You need to configure graceful disconnection to properly shut down applications for deployment, stop, rollback, scale-in, and reset.

Background

Graceful disconnection is a series of operations to ensure that an application is properly shut down before you deploy, stop, roll back, scale in, or reset the application, to avoid data errors, loss, application call exceptions, and other problems caused by unexpected shutdown.

How do we ensure the normal processing of consumer service requests during the period from when applications are stopped to when services are recovered? The most secure and reliable solution is to update your application when no service requests exist. However, in practice, service requests typically exist even when the application is disconnected. A traditional solution is to manually perform three steps: (1) Manually extract traffic. (2) Stop the application. (3) Update your application and then restart it. These manual operations ensure that the update process is imperceptible to clients.

An automatic mechanism can be provided at the container or framework level to manually extract traffic and process received requests. This makes the update process imperceptible to services and improves the O&M efficiency. This mechanism is called graceful disconnection.

Enterprise Distributed Application Service (EDAS) integrates graceful disconnection into the publishing process so that graceful disconnection is automatically implemented when you stop, deploy, roll back, scale in, and reset the applications in Elastic Compute Service (ECS) clusters of EDAS.

Configure graceful disconnection for Dubbo applications

  • Graceful disconnection for services

    By default, graceful disconnection is enabled for Dubbo applications. The default wait time before application shutdown is 10,000 ms. You can modify the wait time by setting dubbo.service.shutdown.wait. For example, set the wait time to 20s by adding the following configuration:

    dubbo.service.shutdown.wait=20000

    Graceful disconnection does not ensure that all sent or received requests have been processed before the application is shut down. To ensure that all requests are properly processed before application shutdown, you can set the preceding parameter based on the processing time of your services.

  • Graceful disconnection for containers

    You can enable graceful disconnection by setting dubbo.shutdown.hook to true when you use Dubbo through org.apache.dubbo.container.Main.

  • QoS-based graceful disconnection

    ShutdownHook does not ensure that all shutdown processes are completed. Dubbo supports multi-step shutdown for full service protection.

    Multi-step shutdown is the stopping of an application in multiple steps. You can run auto O&M scripts or perform manual operations to ensure that all scripts are executed.

    Before an application is shut down, you can disconnect all services by using the QoS command offline. No more requests are sent to the application after services are disconnected from the registry. A wait time is applied to ensure that all existing requests are processed. Then, shut down the application through SIGTERM or SIGINT. This ensures that no services are affected.

    You can use the QoS feature through telnet or HTTP. For more information, see Dubbo-QoS instructions.

Graceful disconnection for EDAS Dubbo applications

The Dubbo-provided graceful disconnection capability works during system runtime only when an O&M system is used with Dubbo. After Dubbo applications are deployed in EDAS, you can manage the lifecycle of Dubbo applications in EDAS. When you stop Dubbo applications in EDAS, you can view the graceful disconnection records in application change orders.
  1. Log on to the EDAS console. In the upper-left corner, select a region.
  2. In the left-side navigation pane, choose Application Management > Applications. On the Applications page, click the name of the Dubbo application in the ECS cluster.
  3. On the Application Details page, perform the following operations to trigger graceful disconnection for Dubbo applications in EDAS.
    • In the upper-right corner, click Stop Application, Deploy Application, or Roll Back Application.
    • After the application is stopped, click Process Instances in Batch on the right side of the page. On the Process Instances in Batch page, select the application that you want to scale in and click Batch Scale In.
    • On the Instance Information tab, find the application instance and click Reset in the Actions column.
  4. In the upper-left corner of the Application Details page, the message A change process is being executed for this application. Status: Executing appears. Click View Details on the right of the message.
  5. On the View Details page, view the change record on graceful disconnection.
    1. Services are deregistered from the Nacos registry during the smooth application disconnection phase, which is displayed in the application change order.
    2. The system checks whether the Dubbo application enables the QoS port when the application is being stopped. If the QoS port is enabled, the system calls the offline command to disconnect services. The application stop process disconnects services after receiving the command.
      2019-08-19 15:01:46.629  INFO 31887 --- [ qos-worker-3-1] o.apache.dubbo.qos.command.impl.Offline  :  [DUBBO] receive offline command, dubbo version: 2.7.3, current host: 30.5.121.2
      2019-08-19 15:01:46.630  INFO 31887 --- [ qos-worker-3-1] o.a.dubbo.registry.nacos.NacosRegistry   :  [DUBBO] Unregister: dubbo://30.5.121.2:20883/com.alibaba.middleware.sp.dubbo.samples.DemoApi? anyhost=true&application=dubbo-nacos-provider-from-d&bean.name=demo1&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=DUBBO&interface=com.alibaba.middleware.sp.dubbo.samples.DemoApi&methods=hi,echo,test1,plus&pid=31887&register=true&release=2.7.3&revision=1.0                               
    Notice
    • kill -9 pid is only used to shut down an application but does not support graceful disconnection. We recommend that you use the application stop function provided by the EDAS console.
    • The wait time set by timeout during graceful disconnection indicates the maximum time for running each destroy rather than the total wait time for all steps. For example, when the wait time is set to 5s, the system waits for 5s when shutting down the server and client, respectively.
    • To enable EDAS to access port 22222 and call the qos command, you must enable the qos command and expose the default port 22222.