Best Practices for Dynamic Configuration with Spring Cloud, Nacos, and KMS

The article introduces how to integrate Nacos with Spring Cloud to achieve dynamic configuration updates and secure sensitive data using KMS without code modification.

By Zunfei Liu

Introduction

The Spring Cloud framework is widely used by developers in the microservice field. @Value is an annotation that every developer will use in practice. In a Spring Bean, the application.properties property can be referenced through the @Value annotation, allowing for configuration code separation and enhancing the flexibility of application code deployment. However, it does not support dynamic configuration updates at runtime. Nacos is a middleware product that integrates service discovery and configuration management. Its configuration center can implement real-time configuration validation at runtime. By configuring the local property files of the project in Nacos, you can easily integrate Nacos into your application. This integration will achieve dynamic configuration updates simply by changing the configurations within the application. The project relies on various properties. Storing some sensitive data in a centralized Nacos may raise security concerns, which Nacos can also address. In this article, we will introduce the above issues.

This article will proceed in the following steps:

• Integrate Nacos to achieve dynamic configuration updates.
• Integrate KMS for zero-code modification to encrypt sensitive configurations.
• Introduce the working principles of Spring Cloud and Nacos.

Regular Configuration Usage in Spring Cloud Applications

In a Spring Cloud application, you can use the @Value annotation in the Bean to reference the property values in the Spring context. You can reference environment variables, JVM parameters, and properties from the commonly used application.properties configuration file.

The following is a simple example of this usage:

application.properties:

app.switch=true
app.threadhold=0.8

A simple Spring Bean:

@Component
public class AppConfig{

  @Value("${app.switch:false}")
  boolean switch;

  @Value("${app.threadhold}")
  double threadhold;

}

AppConfig can be referenced by other Spring Beans and can normally retrieve the app.switch and app.threadhold properties configured in application.properties.

When we need to modify the values of app.switch and app.threadhold, we must modify the content in the configuration file and restart the application. When frequent modifications to certain business parameters are required, restarting the application is relatively inefficient.

Integrate Nacos to Achieve Dynamic Configuration Updates

In this section, we will introduce how to integrate Nacos to achieve dynamic configuration updates in a Spring Cloud application.

Spring has introduced the spring.config.import parameter since version 2.4.x, which can customize external property sources. Through the Spring Cloud Alibaba component, you can add configurations from Nacos as one of Spring's property sources. Therefore, in a Spring Bean, you can also use the @Value annotation to read configurations from Nacos.

Hereafter, we will refer to Spring Cloud Alibaba as SCA.

1. Add SCA Nacos Config dependency to the POM.

<dependency>
      <groupId>com.alibaba.cloud</groupId>
      <artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
      <version>${spring.cloud.alibaba.version}</version>
</dependency>

The version names of the components are related to the Spring Boot versions. You can select the corresponding version based on the version description on the SCA official website: https://sca.aliyun.com/docs/2021/overview/version-explain/

2. Initialize Nacos configurations.

You can select open-source Nacos or purchase a commercial MSE Nacos version. The following figure shows the commercial Nacos.

Assume that our application is for a payment business, and the application is named pay.

In the Nacos instance, you can create configurations of dataId=pay-application.properties and group=core.

3. Modify the application.properties in the application project.

spring.config.import=nacos:pay-application.properties?group=core&refreshEnabled=true
spring.cloud.nacos.config.server-addr={server addr}

Add the spring.config.import parameter to include the configuration from Nacos with dataId=pay-application.properties and group=core as a property source. Setting refreshEnabled=true indicates that when the configuration in Nacos changes, the property source in Spring should be refreshed synchronously.

Add the spring.cloud.nacos.config.server-addr parameter to specify the Nacos address for the connection.

Delete the app.switch and app.threadhold parameters from the local application.properties file in the project.

4. Add the RefreshScope annotation to the Spring Bean.

@Component
@RefreshScope
public class AppConfig{

  @Value("${app.switch:false}")
  boolean switch;

  @Value("${app.threadhold}")
  double threadhold;

}

In the business code, you can still use the @Value annotation to reference configurations from the Spring context. However, you need to add the RefreshScope annotation to the Bean. Adding this annotation will cause Spring to refresh the properties in the current Bean when the internal property source is updated.

After restarting the application, if we modify the properties in the pay-application.properties configuration in Nacos, the parameter values in the AppConfig of the application will be dynamically updated.

Integrate KMS to Encrypt Configurations Imperceptibly

In the previous section, we achieved dynamic configuration updates for Spring Cloud applications by integrating Nacos. The configurations in the application are diverse, some of which are highly sensitive, such as the database connection addresses, usernames and passwords, secret keys of third-party components, tokens, and other sensitive configurations in business functions. The security of these configurations is crucial. If leaked, they could have an immeasurable impact on the business. These data are stored in the pay-application.properties in Nacos. Here is an example:

The sensitive parameters in the following examples are simulated data:

dataId=pay-application.properties,group=core:

# Database Configurations
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/myapp
spring.datasource.username=user001
spring.datasource.password=pass!@#$%

# Secret Keys and Tokens
app.secret=GFYIdryujixxx
key.token=eedsjpp56hko8h

# Business Parameters
app.switch=false
app.threadhold=0.8

Spring Bean:

@Component
public class SecretConfig{

  @Value("${app.secret}")
  String secret;

  @Value("${app.token}")
  String token;

  @PostConstruct
  public void init(){
    //init client use token and secret
  }
}

@Component
public class AppConfig{

  @Value("${app.switch:false}")
  boolean switch;

  @Value("${app.threadhold}")
  double threadhold;

}

There are usually more security considerations for data, such as database passwords and tokens. For example, can the security of these sensitive configurations be guaranteed when stored in Nacos? Is there a possibility of data leakage during the transmission between the application and Nacos? Can different read and write permissions be set for sensitive configurations and regular business configurations? To achieve these security requirements, can the business code be modified at a low cost?

To achieve the above security requirements, we need to do the following:

Sensitive configurations must be stored in Nacos in an encrypted form, not as plain text.
Sensitive configurations need to be encrypted during transmission to prevent intermediate network devices from stealing data by capturing packets.
The application's business code cannot sense whether the configuration is encrypted. The property values still need to be read in the previous way to reduce the modification cost.

The following describes how to achieve the above requirements by integrating KMS for zero-code modification.

Encrypted Configurations Split

If we want to separate regular business configurations from sensitive configurations, we can split the configurations with dataId=pay-application.properties and group=core. We can create a separate encrypted Nacos configuration for sensitive data, such as dataId=cipher-kms-aes-256-pay-application.properties and group=secret, to store sensitive configurations related to data sources and tokens. To prevent duplication of property values between decrypted configuration and regular configuration files, we can prefix all property values in the encrypted configuration with encrypted.

Configurations in Nacos

1. dataId=cipher-kms-aes-256-pay-application.properties, group=secret

# Database Configurations
encrypted.spring.datasource.driver-class-name=com.mysql.jdbc.Driver
encrypted.spring.datasource.url=jdbc:mysql://localhost:3306/mydatabase
encrypted.spring.datasource.username=user001
encrypted.spring.datasource.password=pass!@#$%

# Secret Keys and Tokens
encrypted.app.secret=test_GFYIdryujixxx
encrypted.key.token=test_eedsjpp56hko8h

2. dataId=pay-application.properties, group=core

The properties in the original pay-application.properties should remain unchanged for now. Once all application nodes have integrated the new configuration cipher-kms-aes-256-pay-application.properties, changes can be made to it.

Configuration Modification within the Project

1. Introduce the MSE plug-in extension package.

The encrypted configuration is ciphertext during storage and network transmission. Therefore, you need to support the decryption function on the application side and introduce a decryption plug-in in the POM.

<dependency>
    <groupId>com.alibaba.nacos</groupId>
    <artifactId>nacos-client-mse-extension</artifactId>
    <version>1.0.4</version>
</dependency>

2. Adjust application.properties in the project.

Add a new configuration, cipher-kms-aes-256-pay-application.properties, to the spring.config.import and set the KMS initialization parameters.

spring.config.import=nacos:cipher-kms-aes-256-pay-application.properties?group=secret&refreshEnabled=true,nacos:pay-application.properties?group=core&refreshEnabled=true
spring.cloud.nacos.config.server-addr={server addr}

# Set the parameters required for the NacosClient to access KMS
spring.cloud.nacos.config.kms_region_id=cn-hangzhou
spring.cloud.nacos.config.kmsEndpoint=kst-xxx.cryptoservice.kms.aliyuncs.com
spring.cloud.nacos.config.kmsVersion=v3.0
spring.cloud.nacos.config.kmsClientKeyFilePath=clientKey_hangzhou.json
spring.cloud.nacos.config.kmsPasswordKey=10xxxd1d
spring_cloud_nacos_config_kmsPasswordKey=10xxxd1d
spring.cloud.nacos.config.kmsCaFilePath=clientKey_hangzhou.json

The parameters required for the NacosClient to access KMS are related to the KMS version. For more information, please refer to the MSE official documentation

After you modify the configuration and restart the service application, the application still reads the pay-application.properties values in Nacos, but the properties with the encrypted prefix already exist in the Spring context.

Encrypted Configuration Migration

After the current application was restarted, we modified the Nacos configuration as follows:

dataId=pay-applicaition.properties, core=core

# Database Configurations
spring.datasource.driver-class-name=${encrypted.spring.datasource.driver-class-name}
spring.datasource.url={encrypted.spring.datasource.url}
spring.datasource.username=${encrypted.spring.datasource.username}
spring.datasource.password=${encrypted.spring.datasource.password}

# Secret Keys and Tokens
app.secret=${encrypted.app.secret}
key.token=${encrypted.key.token}

# Business Parameters
app.switch=false
app.threadhold=0.8

The sensitive properties originally in the pay-applicaition.properties reference the property keys in the cipher-kms-aes-256-application.properties by using the ${} mode. The application code does not directly reference the properties in the cipher-kms-aes-256-pay-application.properties. Instead, it indirectly references them through a nested pattern in pay-application.properties. Essentially, the application code still reads the properties from pay-application.properties.

During the process, we only modified the project's configuration files without changing the business code. After the modification, the content in the cipher-kms-aes-256-pay-application.properties configuration is in the ciphertext format on the Nacos server, during transmission, and in the local cache of the application. In the business application process, the NacosClient interacts with KMS to decrypt the ciphertext into plaintext.

Introduction to the Working Principle of Dynamic Configuration Refresh

Through the above modifications, we have completed the function of Spring Cloud + Nacos to achieve dynamic configuration refresh. In this section, we will introduce the working principle of dynamic configuration refresh with Spring Cloud and Nacos.

Startup Loading Mechanism

The initialization of Spring Beans needs to read the configurations in Nacos. Therefore, the Nacos initialization process is performed before all Spring Beans are initialized. The Spring Cloud Alibaba component initializes Nacos based on the current application.properties parameters, loads the configurations from the Nacos server, and builds them into a NacosPropertySource. Spring can also read parameters from the JVM or environment variables at this stage. Therefore, the parameters required for Nacos initialization can also be set through JVM parameters and environment variables, such as the address of the Nacos server, namespace, and authentication-related accessKey and secretKey.

Spring enters the Bean initialization process after the complete property source is built. Only when the initialization of Nacos and the loading of Nacos configurations are completed during this stage can the Bean read the configurations from Nacos.

Dynamic Update Mechanism

In the previous section, when we set the spring.config.import parameter, we specified refreshEnabled=true. This parameter indicates whether to dynamically listen for changes in the configuration on the remote Nacos server. If this parameter is not specified, SCA will only load the configuration once at startup, not monitor the configuration changes and update the contents of the NacosPropertySource during runtime. Thus, the property values in Spring Beans cannot be updated during runtime.

You can understand the mechanism of dynamic Nacos configuration updates by following the numbered sequence in the diagram above. When the refreshEnabled=true parameter is added to the spring.config.import configuration, SCA will listen for the Nacos configurations after the initialization of the Spring container. The timing of configuration monitoring is inconsistent with the timing of configuration startup loading. Configurations are initialized before all Beans are initialized, while the listening for configuration changes occurs after all Beans have completed initialization.

After successful monitoring, when we update the configuration in the Nacos console, the NacosClient in the application will notify SCA of the configuration change. SCA will issue a RefreshEvent to Spring after receiving the callback from the underlying Nacos. The ContextRefresher in Spring will receive the event, update the latest configuration to the NacosPropertySource, update the Environment object, and issue a RefreshScopeRefreshedEvent. All Spring Beans with RefreshScope and ConfigurationProperties annotations are reinitialized, and the latest property values are never obtained.

In the above process, the refreshEnabled=true parameter in the spring.config.import configuration determines whether SCA will listen for the configuration and update the Environment when the configuration in Nacos changes. Adding the RefreshScope and ConfigurationProperties annotations to a Bean determines whether the properties in the Bean will be refreshed when the properties in the Environment object change.

Priority of the Property Sources

As we learned, the @Value annotation can read configurations from environment variables, JVM, and application.properties. Different property sources may have duplicate property keys. In this case, Spring has a priority for reading properties, as shown in the figure below: the priority is JVM parameters > environment variables > Nacos configurations (property sources introduced by the spring.config.import parameter) > local application.properties in the project.

Properties set in Nacos will override those in the local project's property files, but their priority is lower than that of JVM and environment variables. If the same parameters are configured in environment variables and JVM parameters, the configuration in Nacos will not take effect. When implementing dynamic configuration loading, SCA follows the property source priority order officially recommended by Spring Boot. For more information, please refer to https://docs.spring.io/spring-boot/reference/features/external-config.html

In addition, the spring.config.import parameter can specify multiple property sources, and commas "," separate different property sources. Among multiple property sources, those imported earlier have a lower priority.

In versions prior to Spring Boot 2.4, Spring did not support specifying external property sources through spring.config.import. SCA provides spring.cloud.nacos.config.shared-configs and spring.cloud.nacos.config.extension-configs parameters to specify multiple Nacos configuration property sources. In the latest SCA version, 2023.0.1.3, these two parameters have been abandoned and unified to standard spring.config.import parameters. In addition, in the earlier versions of Spring, you can configure parameters in the bootstrap.yml file. This usage is also deprecated in the new version. We recommend that you upgrade the application code that depends on the earlier versions and transform it to a standard method for configuration.

Nacos Logs

Nacos plays a core role in dynamic configuration push. By viewing the startup and runtime logs of Nacos, you can better understand the internal principles of the integration and help you independently troubleshoot common problems in the configuration center. By default, the log directory of the Nacos client is under the {user.home}/logs/nacos/ directory, where {user.home} is the home directory of the user to which the application process runs. In Linux systems, if the process is started as the root user, the logs are located at /root/logs/nacos/. If it is started as the admin user, the logs are located at /home/admin/logs/nacos/. In the Nacos directory, you can see three log files: remote.log, config.log, and naming.log. Among them, remote.log records the logs related to the long connection between the Nacos client and the Nacos server. The naming.log file contains logs related to service management. The config.log file is the core log we need to focus on, as it records detailed logs of interactions between the Nacos client and the Nacos server. Here are a few key logs:

• add-listener: It indicates that the application listens to the configuration, including the namespace,dataId, and group triplet. Only when the application listens to the configuration normally can it receive the push message when the configuration changes.

• server-push: It indicates that the application has received a configuration change push event from the server.

• data-received: It indicates that the application queries the configuration content from the server after receiving the push event, including the namespace, dataId, and group triplet, and the received configuration MD5. You can compare the MD5 value with the Nacos console to determine whether the correct version is received.

• notify-listener: It indicates that the application receives the updated configuration content and attempts to send the latest configuration content to the corresponding listener. For example, it notifies SCA to reload the Nacos configuration and update the Spring context.

• notify-ok: It indicates that Nacos has successfully called back the listener, and the callback in the listener has been executed normally.

• notify-error: It indicates that Nacos calls back the listener, but the listener execution prompts an exception. From the business perspective, this situation shows that the configuration update has no effect and needs to be handled based on the specific exception.

• notify-block-monitor: It indicates that Nacos has called back the listener, but the listener execution times out. By default, the log is printed when the listener execution exceeds 60s.

By reading Nacos logs, you can troubleshoot problems encountered during the Nacos configuration center. For example, you can check whether the application is connected to the correct Nacos server address, whether it listens to the correct namespace, dataId, and group, whether it receives the change push normally, and whether there are abnormal errors and blocking timeouts during listener callback.

During startup and runtime, you can also view the interactions between Nacos and KMS in the config.log file of Nacos to better troubleshoot problems. For more information about how to integrate KMS to achieve configuration encryption and decryption, please refer to the section on Nacos storage security in the article Zero-trust Security Practice of Nacos.

Summary

In the above, we have implemented dynamic configuration updates during runtime in a Spring Cloud application by integrating Nacos. In addition, KMS is used to protect the application's sensitive configurations without changing the code. This solves the data security concerns that may exist when the configuration is migrated to Nacos. The underlying working principle is also briefly introduced. Nacos is a widely used configuration center. In addition to the basic real-time dynamic update of configurations, Nacos also supports advanced features to improve usability, such as configuration listening and querying (configuration subscription node querying), push tracks, and tag-based canary releases.

In terms of security, we split the regular business configurations and sensitive configurations to solve the problem of sensitive data leakage from the application side. In addition, the MSE console provides fine-grained access control functions for different accounts. For example, it allows application developers to access regular business configurations, while sensitive data such as database keys are only accessible to O&M personnel. MSE Nacos also supports this feature. We will have separate articles to introduce the aspects of access control in the future, so please stay tuned for upcoming articles.