During rolling deployments or restarts, upstream services may route requests to downstream instances that are shutting down, causing connection timeouts and errors until the address list is refreshed.
Why traffic errors occur during deployments
When a downstream instance shuts down, a timing gap exists between when it stops accepting requests and when all upstream callers stop sending requests to it. Errors occur during this gap for four reasons:
-
Shutdown before in-flight calls complete. The downstream instance shuts down after a call is initiated but before it sends a response.
-
Slow deregistration due to complex shutdown logic. The downstream instance takes longer to shut down, delaying its deregistration from the registry.
-
Upstream fails to process the updated address list. The downstream instance deregisters as expected, but the upstream service does not promptly process the new address list due to network faults, insufficient resources, or abnormal processing logic.
-
Outdated Nacos client version. An earlier Nacos client version does not promptly remove the IP address of the shut-down downstream instance due to an invalid mechanism.
Recommended solution: graceful rolling deployment
Enable graceful rolling deployment in Microservices Governance to handle graceful release, start, and shutdown automatically.
For setup instructions, see Configure graceful rolling deployment.
Diagnose persistent errors with Nacos client logs
If graceful rolling deployment is not available, diagnose the root cause by analyzing Nacos client logs on the upstream service. Search for the downstream service name and the keyword current ips, then compare three timestamps:
-
Time when the downstream service was changed (deployment or restart initiated)
-
Time of the Nacos client log message (address list update received)
-
Time when the upstream service reported an error
The relationship between these timestamps points to the root cause.
Scenario 1: All three timestamps are closely aligned
The upstream service starts reporting errors after the downstream change begins, then stops after the Nacos client log message appears. This is normal behavior — the errors occur only during the brief propagation delay.
Solution: Use the manual Nacos API approach described in Manual graceful release with Nacos API.
Scenario 2: Downstream change and Nacos log aligned, but errors persist
The Nacos server pushed the updated address correctly, and the Nacos client received it, but the application continues to use the old address list.
Troubleshooting steps:
-
If not using an open source framework, check the application logic for a cache mechanism and verify whether a cache update fails.
-
If using an open source framework, seek help from the corresponding community.
Scenario 3: Downstream change and Nacos log aligned, but slow recovery
The Nacos server pushed the updated address correctly and the Nacos client received it, but the application is slow to adopt the new address list.
Troubleshooting steps:
-
If not using an open source framework, check whether the application has a caching mechanism and whether the cache update is delayed.
-
Check for auxiliary frameworks such as
ribbon,feign, orloadbalance. These frameworks maintain an address list cache that refreshes slowly. Modify the cache refresh configuration for the specific framework. -
If using an open source framework, seek help from the corresponding community.
-
If the issue persists, use the manual Nacos API approach described in Manual graceful release with Nacos API.
Scenario 4: Large gap between downstream change time and the other timestamps
The Nacos client did not detect the downstream service change. This points to a network or version issue.
Troubleshooting steps:
-
Upgrade the Nacos clients for both upstream and downstream services to version 2.x or later.
-
Check whether the upstream service has network faults or insufficient resources.
-
Check for blocking logic in the downstream service shutdown process that prevents the application from responding to requests while it is still registered in the registry.
-
If the issue persists, use the manual Nacos API approach described in Manual graceful release with Nacos API.
Manual graceful release with Nacos API
If graceful rolling deployment is not available, manually control traffic by disabling the downstream instance in the registry before stopping it.
Step 1: Remove the instance from traffic rotation
Before stopping the downstream service, use the Nacos OpenAPI to set enabled=false for the instance, or unpublish the instance in the MSE console. Then confirm through monitoring and logs that the downstream service node no longer receives requests.
Step 2: Stop the service and apply changes
Stop the downstream service node and execute the change (deploy new code, update configuration, etc.).
Step 3: Re-enable the instance
After the change is complete and the node serves traffic correctly, use the Nacos OpenAPI to set enabled=true for the instance, or publish the instance in the MSE console.