Microservices Engine:MSE events reference

Two-tier escalation pattern

Several Nacos events follow a two-tier escalation pattern: a Warning fires first when the system enters monitor mode, then a Critical fires if the condition worsens and the system switches to enforcement mode.

The CloudMonitor event name suffix indicates the current state:

Nacos:TPSRateLimit -- Client TPS throttling

Fires when client transactions per second (TPS) against the Nacos engine exceed the configured threshold.

Remediation:

1. Upgrade the instance specification or scale out nodes to increase capacity.

2. Check whether application code is driving the high TPS -- for example, calling a Nacos API in a tight loop or creating multiple Nacos client instances instead of reusing a singleton.

Nacos:ConfigCapLimit -- Configuration count limit

Fires when the number of configurations in a Nacos instance exceeds the default upper limit of 10,000.

Remediation:

1. Delete unused configurations.

2. Check whether the application is programmatically publishing a large number of configurations -- for example, creating one configuration per request instead of aggregating values.

Nacos:ServiceCapLimit -- Service registration limit

Fires when the number of registered services reaches the instance storage upper limit.

Remediation:

1. Upgrade the instance specification or scale out nodes.

2. Check for invalid or duplicate service registrations, or multiple Nacos client instances creating redundant registrations.

Nacos:ConnectionCapLimit -- Connection count limit

Fires when the number of connected clients reaches the instance upper limit.

Remediation:

1. Upgrade the instance specification or scale out nodes.

2. Check for multiple Nacos client instances per application or connection leaks caused by clients that are not properly closed.

Nacos:ParametersIllegalDenied -- Invalid request parameters

Fires when a client sends invalid request parameters to the Nacos instance.

Remediation:

Check whether request parameters of the client are valid.

Nacos:CPUAlert -- High CPU load

Fires when CPU load on a Nacos instance is too high and the instance is overloaded. Subsequent requests may respond slowly or fail.

Remediation:

1. Upgrade the instance specification or add nodes to distribute load.

2. Check whether the high CPU load is caused by improper use of the instance.

Nacos:MemoryAlert -- High memory usage

Fires when memory usage on a Nacos instance is too high. An out-of-memory (OOM) killer may terminate processes, causing instance nodes to restart.

Remediation:

Upgrade the instance specification. High memory usage typically indicates that the instance is handling more services, configurations, or connections than its current specification supports.

Nacos:FullGc -- Frequent full garbage collections

Fires when frequent full garbage collections (GCs) reduce available capacity. Subsequent requests may respond slowly or fail.

Remediation:

1. Upgrade the instance specification or add nodes.

2. Check whether the capacity is insufficient due to improper use of the instance.

Nacos:TooManyConfigAlert -- Configuration count exceeded

Fires when the number of configurations exceeds 10,000. Additional configurations cannot be created, and new configuration releases may fail.

Remediation:

Delete unnecessary configurations.

Nacos:TooManyConfigLongPolling -- Excessive configuration listeners

Fires when too many listeners are configured for a Nacos instance, causing overload. Frequent full GCs may follow, leading to slow or failed responses.

Remediation:

1. Upgrade the instance specification or add nodes.

2. Review application code for unnecessary configuration listeners. Each listener maintains a long-polling connection, so reducing listener count directly lowers resource consumption.

Nacos:ProviderDropTooFast -- Sudden decrease in service providers

Fires when the number of registered service providers drops by more than 50% compared to 3 minutes ago. Upstream services may fail to discover downstream providers.

Remediation:

1. Check for widespread application failures or network issues that caused multiple providers to deregister simultaneously.

2. If the drop resulted from a planned large-scale deployment or release, this event can be safely ignored.

ZooKeeper:ExceedEphemeralLimit -- Ephemeral node limit exceeded

Fires when a single session creates more than 2,000 ephemeral nodes. Additional ephemeral nodes may fail to be added.

Remediation:

1. Review business code for excessive ephemeral node creation within a single session.

2. If you use Dubbo, check the version. Older Dubbo versions may create excessive ephemeral nodes. Upgrade to the latest version.

3. Add the Jute.Maxbuffer parameter to ensure successful recovery from failures.

ZooKeeper:FullGc -- Long full garbage collections

Fires when full GCs take too long, increasing processing latency and causing response timeouts. ZooKeeper instance nodes may be recovered from a failure and restarted.

Remediation:

1. Check the ZooKeeper version and upgrade if outdated. Newer versions include GC optimizations.

2. If this event persists, add nodes to the instance to increase capacity.

ZooKeeper:CPUAlert -- High CPU load

Fires when CPU load is too high, increasing processing latency and causing response timeouts. ZooKeeper instance nodes may be recovered from a failure and restarted.

Remediation:

If this event persists, add nodes to the instance to increase capacity.

Gateway:ElasticScaleOut -- Elastic scale-out

Gateway:ElasticScaleIn -- Elastic scale-in

Gateway:ElasticOpen -- Enable elastic scale-out

Gateway:ElasticClose -- Disable elastic scale-out

Gateway:ElasticStrategyUpdate -- Elasticity policy update

Governance:FlowTrafficBlock -- Throttling rule triggered

Fires when an interface throttling rule configured in Microservices Governance is triggered.

Remediation:

Burst traffic is hitting the interface. Monitor traffic trends and assess whether the throttling threshold needs adjustment.

Governance:CircleBreakerTrafficBlock -- Circuit breaking rule triggered

Fires when a circuit breaking rule configured in Microservices Governance is triggered.

Remediation:

Check downstream interface dependencies for unstable performance, such as slow calls or service errors.

Governance:ParamsTrafficBlock -- Hotspot parameter protection triggered

Fires when a hotspot parameter protection rule configured in Microservices Governance is triggered.

Remediation:

If hotspot parameters are called frequently, prefetch the relevant data to reduce per-request load.

Lifecycle events

The following Info-level events track microservice lifecycle operations. No action is required unless the event is unexpected.