Event Center in the Microservices Engine (MSE) console collects event data that is generated by Microservices Registry, Microservices Governance, and cloud-native gateways. You can view event changes of all resources managed in MSE and troubleshoot issues based on abnormal events. This topic describes the events that may be generated in MSE and provides suggestions on how to handle each event.
Events of Nacos
Event code | Event name | Event severity | CloudMonitor event name | Event description and impact | Event handling suggestion |
Nacos:TPSRateLimit | Nacos Client TPS Throttling | Warning | Nacos:TPSRateLimit:Normal | When the client accesses the Nacos engine, the transactions per second (TPS) is excessively high. To ensure stable operations of the Nacos engine, the system is about to perform throttling on requests. In this case, the Nacos instance is in the monitor mode. When the TPS threshold is reached, subsequent requests are still allowed. | Upgrade the instance configuration or scale out nodes, and check whether the high TPS is caused by incorrect usage. For example, a Nacos API operation is frequently called or a Nacos client is created multiple times. |
Critical | Nacos:TPSRateLimit:Executing | When the client accesses the Nacos engine, the TPS is excessively high. To ensure stable operations of the Nacos engine, the system performs throttling on requests. When the TPS threshold is reached, subsequent requests are rejected. | Upgrade the instance configuration or scale out nodes, and check whether the high TPS is caused by incorrect usage. For example, a Nacos API operation is frequently called or a Nacos client is created multiple times. | ||
Nacos:ConfigCapLimit | Nacos Configuration Limiting | Warning | Nacos:ConfigCapLimit:Normal | The number of configurations in a Nacos instance exceeds the default upper limit 10,000. The system is about to perform throttling on new configurations. In this case, the Nacos instance is in the monitor mode. New configurations can still be created and released. | Delete unused configurations at the earliest opportunity and check whether a large number of configurations are automatically released by the application. |
Critical | Nacos:ConfigCapLimit:Executing | The number of default configurations in a Nacos instance exceeds 10,000, and the system no longer allows users to create configurations. As a result, new configurations cannot be released. | Delete unused configurations at the earliest opportunity and check whether a large number of configurations are automatically released by the application. | ||
Nacos:ServiceCapLimit | Nacos Service Limiting | Warning | Nacos:ServiceCapLimit:Normal | An excessive number of services are registered, and the storage upper limit is reached. To ensure stable operations of the Nacos engine, the system is about to perform throttling on service registrations. In this case, the Nacos instance is in the monitor mode. New services or service providers can still be registered with the Nacos instance. | Upgrade the instance configuration or scale out nodes, and check whether invalid service publish occurs, repeated service publish occurs, or a Nacos client is created multiple times. |
Critical | Nacos:ServiceCapLimit:Executing | An excessive number of services are registered, and the storage upper limit is reached. To ensure stable operations of the Nacos engine, the system no longer allows service registration. As a result, new services or service providers cannot be registered with the Nacos engine. | Upgrade the instance configuration or scale out nodes, and check whether invalid service publish occurs, repeated service publish occurs, or a Nacos client is created multiple times. | ||
Nacos:ConnectionCapLimit | Nacos Connection Limiting | Warning | Nacos:ConnectionCapLimit:Normal | The number of clients that are connected to the Nacos engine reaches the upper limit of the instance. To ensure stable operations of the Nacos engine, the system is about to perform throttling on new connections. In this case, the Nacos instance is in the monitor mode. New Nacos clients can still be connected to the Nacos engine. | Upgrade the instance configuration or scale out nodes, and check whether incorrect usage exists. For example, the Nacos client is created multiple times, or connection leaks occur due to the Nacos client enabled. |
Critical | Nacos:ConnectionCapLimit:Executing | The number of clients that are connected to the Nacos engine reaches the upper limit of the instance. To ensure stable operations of the Nacos engine, the system no longer allows the establishment of new connections. As a result, new Nacos clients cannot be connected to the Nacos engine. | Upgrade the instance configuration or scale out nodes, and check whether incorrect usage exists. For example, the Nacos client is created multiple times, or connection leaks occur due to the Nacos client enabled. | ||
Nacos:ParametersIllegalDenied | Invalid Request Parameters for Nacos Instances | Warning | Nacos:ParametersIllegalDenied:Normal | The request parameters sent by the client to the Nacos instance are invalid. The server is about to reject invalid parameters. In this case, the Nacos instance is in the monitor mode. Normal results can still be returned. | Check whether request parameters of the client are valid. |
Critical | Nacos:ParametersIllegalDenied:Executing | The request parameters sent by the client to the Nacos instance are invalid. The server rejects the invalid parameters and returns an error code. | Check whether request parameters of the client are valid. | ||
Nacos:CPUAlert | Excessively High CPU Load in Nacos Instances | Warning | Nacos:CPUAlert:Executing | The CPU load of a Nacos instance is excessively high and the instance is overloaded. This may cause slow responses or even no responses to subsequent requests. | Upgrade the instance configuration or add instance nodes, and check whether the high CPU load is caused by improper use of the instance. |
Nacos:MemoryAlert | Excessively High Memory Usage in Nacos Instances | Warning | Nacos:MemoryAlert:Executing | The memory usage of a Nacos instance is excessively high, which may cause an out of memory (OOM) killer to run. This may cause the nodes of the Nacos instance to restart. | Upgrade the instance configuration. |
Nacos:FullGc | Full GCs of Nacos Instances | Warning | Nacos:FullGc:Executing | The capacity of a Nacos instance becomes insufficient due to frequent full garbage collections (GCs). This may cause slow responses or even no responses to subsequent requests. | Upgrade the instance configuration or add instance nodes, and check whether the capacity is insufficient due to improper use of the instance. |
Nacos:TooManyConfigAlert | Excessive Configurations for Nacos Instances | Warning | Nacos:TooManyConfigAlert | The number of configurations that you attempt to create for a Nacos instance exceeds 10,000. The extra configurations cannot be created. New configurations may fail to be released. | Delete unnecessary configurations. |
Nacos:TooManyConfigLongPolling | Excessive Long Pollings Configured for Nacos Instances | Warning | Nacos:TooManyConfigLongPolling | The number of listeners configured for a Nacos instance is excessively large, which results in an overload issue. Full GCs may be frequently triggered, which may cause slow responses or even no responses to subsequent requests. | Upgrade the instance configuration or add instance nodes, and check whether the capacity is insufficient due to improper use of the instance. |
Nacos:ProviderDropTooFast | Sudden Decrease of Providers on Nacos Instances | Warning | Nacos:ProviderDropTooFast:Executing | The number of service providers that are registered with a Nacos instance at the current time is decreased by more than 50% compared with 3 minutes ago. When this event is generated, the upstream service may fail to identify the downstream service providers. | Check whether a large number of applications fail or network failures occur. If the event is generated due to the expected release of a large number of applications, you can ignore the event. |
Events of Zookeeper
Event code | Event name | Event severity | CloudMonitor event name | Event description and impact | Event handling suggestion |
ZooKeeper:ExceedEphemeralLimit | Ephemeral Nodes Created by a Single Session in ZooKeeper Exceed Limit (Limit = 2000) | Critical | ZooKeeper:ExceedEphemeralLimit | The number of ephemeral nodes that are created by a single session is excessively large. This may cause a failure to add ephemeral nodes to the session. |
|
ZooKeeper:FullGc | Full GCs of ZooKeeper Instances | Warning | ZooKeeper:CMSGc:Executing | Full GCs take a long time. This may increase processing latency and cause response timeouts. As a result, nodes of the ZooKeeper instance are recovered from a failure and restarted. |
|
ZooKeeper:CPUAlert | Excessively High CPU Load in ZooKeeper Instances | Warning | ZooKeeper:CPUAlert:Executing | The CPU load is excessively high. This may increase processing latency and cause response timeouts. As a result, nodes of the ZooKeeper instance are recovered from a failure and restarted. | This event is generated due to insufficient capacity. If this event persists, we recommend that you add nodes to the instance. |
Events of cloud-native gateways
Event code | Event name | Event severity | CloudMonitor event name | Event description and impact | Event handling suggestion |
Gateway:ElasticScaleOut | Elastic Scale-out of Cloud-native Gateways | Warning | Gateway:ElasticScaleOut:Failed | When an elastic scale-out operation is performed on a cloud-native gateway, the cloud-native gateway generates this event based on the scale-out status. | Handle the event based on the event details in the console. |
Info | Gateway:ElasticScaleOut:Executing | Take note of the event information if necessary. | |||
Info | Gateway:ElasticScaleOut:Executed | Take note of the event information if necessary. | |||
Gateway:ElasticScaleIn | Elastic Scale-in of Cloud-native Gateways | Warning | Gateway:ElasticScaleIn:Failed | When an elastic scale-in operation is performed on a cloud-native gateway, the cloud-native gateway generates this event based on the scale-in status. | Handle the event based on the event details in the console. |
Info | Gateway:ElasticScaleIn:Executing | Take note of the event information if necessary. | |||
Info | Gateway:ElasticScaleIn:Executed | Take note of the event information if necessary. | |||
Gateway:ElasticOpen | Enable Elastic Scale-out of Cloud-native Gateways | Warning | Gateway:ElasticOpen:Failed | When you attempt to enable elastic scale-out, cloud-native gateways generate this event based on the enabling result. | Handle the event based on the event details in the console. |
Info | Gateway:ElasticOpen:Executed | Take note of the event information if necessary. | |||
Gateway:ElasticClose | Disable Elastic Scale-out of Cloud-native Gateways | Warning | Gateway:ElasticClose:Failed | When you attempt to disable elastic scale-out, cloud-native gateways generate this event based on the disabling result. | Handle the event based on the event details in the console. |
Info | Gateway:ElasticClose:Executed | Take note of the event information if necessary. | |||
Gateway:ElasticStrategyUpdate | Elasticity Policy Update of Cloud-native Gateways | Warning | Gateway:ElasticStrategyUpdate:Failed | When elastic scale-out is enabled, cloud-native gateways update the elastic scale-out policy and generate this event based on the update result. | Handle the event based on the event details in the console. |
Info | Gateway:ElasticStrategyUpdate:Executed | Take note of the event information if necessary. |
Microservices Governance
Event code | Event name | Event severity | CloudMonitor event name | Event description and impact | Event handling suggestion |
Governance:FlowTrafficBlock | Throttling Rules Triggered for Applications | Warning | Governance:FlowTrafficBlock:Executing | This event is generated when the interface throttling rule configured for the application that is enabled with Microservices Governance is triggered. | Burst traffic occurs on the interface. We recommend that you continue to observe the traffic. |
Governance:CircleBreakerTrafficBlock | Circuit Breaking Rules Triggered for Applications | Warning | Governance:CircleBreakerTrafficBlock:Executing | This event is generated when the circuit breaking rule configured for the application for which Microservices Governance is enabled is triggered. | If the performance of the downstream interface is unstable, such as slow calls and abnormal services, we recommend that you check dependencies of the downstream interface. |
Governance:ParamsTrafficBlock | Hotspot Parameter Protection Rules Triggered for Applications | Warning | Governance:ParamsTrafficBlock:Executing | This event is generated when the hotspot parameter protection rule configured for the application that is enabled with Microservices Governance is triggered. | If hotspot parameters are called, we recommend that you prefetch the relevant data. |
Governance:GracefulShutdown | Graceful Shutdown of Microservices | Info | Governance:GracefulShutdown | This event is generated when a microservice application is gracefully shut down. After Microservices Governance is enabled for the application, the graceful shutdown feature is automatically enabled. | Take note of the event information if necessary. |
Governance:RegistrationFinish | Microservices Registration Finished | Info | Governance:RegistrationFinish | This event is generated when a microservice application is registered and started after the graceful start feature is enabled. | Take note of the event information if necessary. |
Governance:WarmupStart | Microservices Prefetching Started | Info | Governance:WarmupStart | This event is generated when a new instance of the application enabled with Microservices Governance starts to prefetch traffic after the graceful startup feature is enabled. | Take note of the event information if necessary. |
Governance:WarmupEnd | Microservices Prefetching Completed | Info | Governance:WarmupEnd | This event is generated when traffic of new instances of the application enabled with Microservices Governance is completely prefetched after the graceful startup feature is enabled. | Take note of the event information if necessary. |
Governance:ReadinessFinish | Microservice Application Passed Kubernetes Readiness Check | Info | Governance:ReadinessFinish | This event is generated when a Kubernetes readiness probe is completed for the application enabled with Microservices Governance after the graceful startup feature is enabled. | Take note of the event information if necessary. |
Governance:GracefulShutdown | Manually Disable Microservices | Info | Governance:GracefulShutdown | This event is generated when an application instance is manually shut down on the Node details page of Microservices Governance and the operation takes effect. | Take note of the event information if necessary. |
Governance:RegistrationFinish | Manually Enable Microservices | Info | Governance:RegistrationFinish | This event is generated when an application instance is manually started on the Node details page of Microservices Governance and the operation takes effect. | Take note of the event information if necessary. |
Governance:ApplicationStop | Microservice Application Shutdown | Info | Governance:ApplicationStop | This event is generated when the graceful shutdown process is completely performed on the application enabled with Microservices Governance after the graceful shutdown feature is enabled and before the application stops. | Take note of the event information if necessary. |