When a service provider instance becomes unhealthy due to a memory leak or similar issue, it can destabilize the entire microservices system. Microservices Engine (MSE) Microservices Governance lets you isolate the abnormal instance by removing it from Microservices Registry, which stops new traffic from reaching it. After isolation, you can create a memory snapshot through Application Real-Time Monitoring Service (ARMS) to capture the runtime state for root-cause analysis.
The workflow has two phases:
-
Isolate : Take the abnormal instance offline to stop incoming traffic.
-
Diagnose : Create a memory snapshot of the isolated instance while the fault state is still intact.
Prerequisites
Before you begin, make sure that you have:
-
A Java application that uses Dubbo 2.6.x, Dubbo 2.7.x, or Spring Cloud Edgware and later
-
The target instance connected to ARMS. For setup instructions, see Integration Center
If
spring.cloud.xxxx.discovery.fail-fast
is set to
false
for a Spring Cloud application, MSE cannot determine whether the application is online or offline.
Step 1: Take the abnormal instance offline
-
Log on to the MSE console and select a region in the top navigation bar.
-
In the left-side navigation pane, choose Microservices Governance > Application Governance .
-
On the Application list page, click the resource card of the target application.
-
In the left-side navigation pane, click Node details .
-
Find the abnormal instance and click Service offline in the Actions column.
-
In the Microservice offline message, click OK .
The instance is removed from Microservices Registry. If no new requests are sent to the instance, you can proceed to create a memory snapshot for troubleshooting.
Step 2: Create a memory snapshot
After the instance is taken offline, create a memory snapshot to record its runtime state for diagnostics.
-
On the Node details page, find the isolated instance and click To create a memory snapshot in the Actions column.
-
In the Hint message, click OK . You are redirected to the ARMS console.
-
On the Application Details page in the ARMS console, click Create Memory Snapshot in the upper-right corner of the JVM monitoring tab.
-
In the Create Memory Snapshot dialog box, click Save .
The memory snapshot records the runtime state of the abnormal instance for analysis.
References
For more information about how to create and view memory snapshots, see Memory snapshots .