All Products
Search
Document Center

Microservices Engine:Empty list protection

Last Updated:Feb 28, 2024

A consumer subscribes to the list of providers from a registry. If a registry changes or encounters an unexpected issue or the connection between a provider and a registry is disconnected due to factors such as networking and CPU loads, subscription exceptions may occur. As a result, the consumer may obtain an empty list of providers. You can enable empty list protection on the Nacos client or Microservices Engine (MSE) Nacos server to improve the availability of the entire system.

Prerequisites

Enable empty list protection on the client

Limits

The Nacos Java client in 1.4.1 or later supports empty list protection.

Note

To prevent risks that arise from specific versions, we recommend that you select appropriate versions for the client, Spring Cloud applications, and Dubbo applications based on the recommended versions.

Procedure

  • If you use the Nacos Java client, perform the following operations:

    1. Add the following dependency of the Nacos client:

      <!-- ${nacos-client.version} must be 1.4.1 or later. -->
      <dependency>
          <groupId>com.alibaba.nacos</groupId>
          <artifactId>nacos-client</artifactId>
          <version>${nacos-client.version}</version>
      </dependency>
    2. Configure the following settings in the application code:

      Properties properties = new Properties();
      properties.put(PropertyKeyConst.SERVER_ADDR, "${Endpoint of the MSE Nacos instance}");
      properties.put(PropertyKeyConst.NAMING_PUSH_EMPTY_PROTECTION, "true");
      NamingService naming = NamingFactory.createNamingService(properties);
  • If your application is a Spring Cloud Alibaba application, add the following configuration to the configuration file of your application:

    spring.cloud.nacos.discovery.namingPushEmptyProtection=true
  • If your application is a Dubbo application, add the following registry URL configuration to the configuration file of your Dubbo application:

    dubbo.registry.address=nacos://${Endpoint of the MSE Nacos instance}:8848?namingPushEmptyProtection=true

Enable empty list protection on the server

Note

A large number of non-Java clients and Nacos Java clients in versions earlier than 1.4.1 are still in use. MSE optimizes the capabilities of Nacos servers based on open source Nacos to provide empty list protection for Nacos servers.

Limits

The Nacos engine version must be 2.1.0.0 or later. For more information about how to upgrade the Nacos engine version, see Update a Nacos version.

Procedure

The method to enable empty list protection varies based on the version of your MSE Nacos engine.

  • If the version of your MSE Nacos engine is 2.1.0.0 or 2.1.0.1, no operation is required. Empty list protection is automatically enabled in these versions.

  • If the version of your MSE Nacos engine is 2.1.0.2 or later, perform the following steps to enable empty list protection:

    1. Log on to the MSE console and select a region in the top navigation bar.

    2. In the left-side navigation pane, choose Microservices Registry > Instances.

    3. On the Instances page, click the name of the instance.

    4. In the left-side navigation pane, click Parameter Settings. In the Real-time Effective Parameters section of the Parameter Settings page, click Edit in the Actions column of the forcePushEmptyProtectionForAllService parameter.

    5. In the Edit Parameters dialog box, select Yes for Value and click OK.

      Note

      To ensure the highest service availability, we recommend that you select Yes for Value of the forcePushEmptyProtectionForAllService parameter in all cases.

Verify the result

When a service triggers empty list protection, Trigger push empty protection for Service is displayed in the naming.log file in the ${user.home}/logs/nacos directory of the node where the consumer resides.

In normal cases, such information does not frequently appear. If such information frequently appears, you can troubleshoot this issue based on the following possible causes:

  • The providers to which the consumer subscribes do not exist.

    If the consumer subscribes to a non-existing provider, you must check whether the application dependency is required. If the application dependency is not required, delete the invalid dependency. If the application dependency is required, you must publish the provider again.

  • No providers are displayed in the instance list.

    • Check whether the providers to which the consumer subscribes are frequently disconnected.

    • Check whether the network connection between each provider and the MSE Nacos instance is normal. If the network connection is abnormal, the MSE Nacos instance determines that the provider is disconnected.

Identify the service that triggers empty list protection

The error message contains the information about the service that triggers empty list protection. You can identify the empty service to which the consumer subscribes based on Service{namespace='XXX', group='xx', name='xxxxxxx', ephemeral=true, revision=0}.

Known trigger scenarios

Scenario

Cause

Solution

In multi-subscription scenarios in which Dubbo 2 versions are compatible, the providers whose service names are in the old format are not displayed in the instance list.

The service name format of a provider in a version earlier than Dubbo 2.7.6 is different from the service name format of a provider in Dubbo 2.7.6 or later. In multi-subscription scenarios in which providers in Dubbo 2.7.6 or later are used, the consumer subscribes to both the providers whose service names are in the old format and the providers whose service names are in the new format. If the versions of all providers are Dubbo 2.7.6 or later, the providers whose service names are in the old format cannot be registered with the Nacos instance. As a result, empty list protection is triggered when the consumer subscribes to a provider whose service name is in the old format.

Upgrade your engine version to 2.1.0.1 or later, or upgrade the Dubbo version to 2.7.17 or later. For more information, see Upgrade an engine version.

In multi-subscription scenarios in which Dubbo 3 versions are compatible, the providers whose service names are interface-level service names are not displayed in the instance list.

Dubbo 3 supports application-level service discovery. The registered service names are not interface-level service names but application-level service names. To ensure smooth upgrades of Dubbo versions, the consumer subscribes to both providers whose service names are application-level service names and providers whose service names are interface-level service names. If the versions of all providers are Dubbo 3, the providers whose service names are interface-level service names cannot be registered with the Nacos instance. As a result, empty list protection is triggered when the consumer subscribes to a provider whose service name is an interface-level service name.

Add the configuration dubbo.application.service-discovery.migration=FORCE_APPLICATION. For more information, visit the Dubbo official website.

If the NacosWatch feature is enabled for a Spring Cloud Alibaba application for the first time and no instance replica exists, an empty instance list is received by the consumer.

The NacosWatch feature is provided in the new version of Spring Cloud Alibaba applications to monitor the service status of the applications. The NacosWatch feature allows a Spring Cloud Alibaba application to monitor its service when the application is started. If the NacosWatch feature is enabled for a Spring Cloud Alibaba application for the first time and no instance replica exists, empty list protection is triggered because the application instance is not registered with the Nacos instance. After the application instance starts, empty list protection is no longer triggered.

Ignore the issue if the NacosWatch feature is enabled for a Spring Cloud Alibaba application for the first time and no instance replica exists. You can also add the configuration spring.cloud.nacos.discovery.watch.enabled=false to disable the NacosWatch feature. For more information, see Spring Cloud Alibaba.

In scenarios in which a service is completely disabled for a Spring Cloud Gateway, the service status is not synchronized.

When the Spring Cloud Gateway is started, the Spring Cloud Gateway queries all services from the Nacos instance and subscribes to all services. For example, after a Spring Cloud Gateway is started, a service is completely disabled because the Nacos instance automatically removes all application instances. However, the Spring Cloud Gateway does not detect the service status and continues to subscribe to the service. As a result, empty list protection is triggered.

The Spring Cloud Gateway cannot dynamically detect the status of a service and unsubscribe from the service if the service is disabled. We recommend that you restart the Spring Cloud Gateway.

Scenarios in which empty list protection is forcefully enabled

In MSE Nacos 2.1.0.2 or later, the logic for enabling empty list protection is optimized. This optimization ensures that empty list protection is enabled only if Yes is selected for Value of the forcePushEmptyProtectionForAllService parameter. However, the system still automatically enables empty list protection to ensure availability and stability in the following scenarios even if No is selected for Value of the forcePushEmptyProtectionForAllService parameter.

  • When an engine node is upgraded, restarted, or recovered from a failure, the node automatically enables two-minute empty list protection for all applications to ensure service availability during the period of time.

  • If all instances of an application are removed from the application instance list on the MSE instance, the MSE instance automatically enables one-minute empty list protection for the application. This operation prevents empty list protection from being unexpectedly triggered when issues such as unstable network performance and temporary failures of service providers occur. This way, service availability is provided in this scenario.

Note

To ensure the highest service availability, we recommend that you select Yes for Value of the forcePushEmptyProtectionForAllService parameter to enable empty list protection in all cases.