Troubleshooting application connection timeouts with MSE Nacos - Microservices Engine

Diagnose and resolve the connection timeouts that your application reports when it connects to MSE Nacos. The checks are organized by the scope and the frequency of the errors, so confirm both before you start.

Symptoms

When your application times out while it connects to MSE Nacos, it reports one of the following errors:

Connection timed out
Read Timeout
TimeoutException: Waited 3000 milliseconds

Possible causes

A connection timeout between your application and MSE Nacos has one of the following causes:

Network transmission failure — Requests that your client sends do not reach the MSE Nacos instance, or responses from the instance do not reach your client. Alternatively, the instance processes requests too slowly, and the request times out.
Public endpoint blocked — An ACL policy blocks the public endpoint of the MSE Nacos instance.
VPN interference — The VPN that your client uses causes network problems.
Client-side resource exhaustion — The processing threads of your client are blocked or fail, or your client is in a state such as full GC, out-of-memory (OOM), or CPU contention. Your client cannot process the data packets that the instance returns in time, and it treats the request as timed out.

Diagnose and resolve the timeout errors

Each branch starts with the condition that it applies to. Work through the branches in order, complete the checks in every branch whose condition matches your case, and continue until the errors stop. The first five branches locate the cause. The last branch increases the timeout period on the client, which reduces the impact of the errors but does not remove their cause.

Timeout errors occur on only one client node. The network between that node and the MSE Nacos instance might be faulty, or the node itself might be faulty or blocked. Both a network transmission failure and client-side resource exhaustion produce this pattern.
On the node where the errors occur, run the ping, telnet, and curl commands to access the MSE Nacos instance:
```
ping ${mse.nacos.host}
telnet ${mse.nacos.host} 8848
curl ${mse.nacos.host}:8848/nacos/v1/ns/service/list
```
${mse.nacos.host} with the MSE Nacos endpoint that your application is configured to use. The results of these commands show whether a network problem exists between the node and the instance.
Then check the monitoring data of the client node for high CPU utilization, frequent full GCs, and OOM errors. These metrics describe the client itself. If any of them is abnormal, the node cannot process the packets that the instance returns in time, which points to client-side resource exhaustion rather than to the network.
Your application connects over the public endpoint. Check whether an ACL policy blocks the public endpoint.
For instructions on how to check whether an ACL policy blocks a public endpoint, see Configure a public IP address whitelist.
Your application connects over the internal endpoint. Check whether your application is deployed in the same region and the same VPC as the MSE Nacos instance. If the application is deployed in a different region or belongs to a different VPC, deploy it in the region or the VPC where the MSE Nacos instance resides. Alternatively, connect the two VPCs through Cloud Enterprise Network (CEN). For more information about CEN, see What is CEN?.
The Basic Information page of the MSE console shows the region where the MSE Nacos instance resides and the VPC ID that corresponds to its internal endpoint.
Your client uses a VPN. Disable the VPN or check the VPN settings, and then connect again.
Timeout errors occur on all client nodes. In the MSE console, go to the Monitoring Center page and check the following metrics of the MSE Nacos instance:
- On the Overview tab, check whether the queries per second (QPS) or the operations per second (OPS) of the instance exceeds the maximum TPS of your instance specifications.
  For the maximum TPS of each instance specification, see Estimate the capabilities of common instances.
- On the Number of Connections Monitoring tab, check whether the number of clients or the number of persistent connections exceeds the maximum number of connections that your instance specifications support.
  For the maximum number of connections of each instance specification, see Estimate the capabilities of common instances.
- On the jvm Monitoring tab, check whether full GC occurs frequently on the instance.
  Note
  On this tab, No data for full GC indicates that full GC has never been performed.
- If your application connects over the public endpoint, go to the Resource Monitoring tab and check whether the inbound traffic and the outbound traffic of the instance exceed the bandwidth that you specified when you purchased the instance.
- On the Resource Monitoring tab, check whether the Memory Usage and the vCPU Utilization of the instance approach or exceed 100%. A load that exceeds the capacity of the instance causes requests to be throttled. In this case, upgrade the instance specifications to increase capacity.
  For instructions on how to upgrade the instance specifications, see Change instance specifications.
For more information about how to view the monitoring data of MSE Nacos, see Monitor engines.
Timeout errors occur only occasionally. Set a longer timeout period so that occasional errors do not affect your application. The parameters to set depend on the version of your Nacos Java client. In the following parameter sets, replace <milliseconds> with the timeout period that you want to apply.
- If your Java client version is 1.0.0 to 1.4.X, add the following parameters to the Java Virtual Machine (JVM) parameters of the application process:
```
-D com.alibaba.nacos.client.naming.ctimeout=<milliseconds>
-D com.alibaba.nacos.client.naming.rtimeout=<milliseconds>
-D NACOS.CONNECT.TIMEOUT=<milliseconds>
```
  - com.alibaba.nacos.client.naming.ctimeout — Connection timeout for the registry, in milliseconds. Default value: 3000.
  - com.alibaba.nacos.client.naming.rtimeout — Request timeout for the registry, in milliseconds. Default value: 50000.
  - NACOS.CONNECT.TIMEOUT — Connection timeout for the configuration center, in milliseconds. Default value: 1000.
- If your Java client version is 2.0.0 to 2.1.1, upgrade the Java client to 2.1.2 or later, and then set the timeout parameters that apply to version 2.1.2 and later.
- If your Java client version is 2.1.2 or later, add the following parameters to the JVM parameters of the application process:
```
-Dnacos.remote.client.grpc.timeout=<milliseconds>
-Dnacos.remote.client.grpc.server.check.timeout=<milliseconds>
-Dnacos.remote.client.grpc.health.timeout=<milliseconds>
```
  - nacos.remote.client.grpc.timeout — Request timeout, in milliseconds. Default value: 3000.
  - nacos.remote.client.grpc.server.check.timeout — Server health check timeout, in milliseconds. The check determines whether the connected server is healthy. If the server is unhealthy, a reconnection is triggered. Default value: 3000.
  - nacos.remote.client.grpc.health.timeout — Connection health check timeout, in milliseconds. The check determines whether the current connection is healthy. If the connection is unhealthy, a reconnection is triggered. Default value: 3000.