This topic describes the issue of a continuous increase in the memory usage of sidecars, and provides possible causes and solutions.

Issue description

During the memory monitoring for pods, it is found that the memory usage of sidecars keeps increasing.

Causes

The following table describes the possible causes of a continuous increase in the memory usage of sidecars.
CauseDescription
Configurations stored in sidecarsBy default, sidecars store the configurations of all services in clusters, such as endpoint information. The configurations occupy a certain amount of memory space. With more and more services deployed in clusters, the service configurations increase and therefore occupy more memory space.
Traffic of real-time requestsAs a forwarding proxy, Envoy has buffers for storing received messages and response messages and caches queued requests. The buffers and cache occupy a certain amount of memory space.
Metrics with too many statistical valuesMetrics with too many statistical values occupy a large amount of memory space.
gRPC throttling-related dataThe HTTP/2 codec of sidecars has the initial_stream_window_size and initial_connection_window_size parameters. The initial_stream_window_size parameter indicates the initial limit on the number of bytes that may be buffered by a single stream in the HTTP/2 codec buffers. The initial_connection_window_size parameter indicates the initial limit on the number of bytes that may be buffered by the connection across all streams in the HTTP/2 codec buffers. By default, the two parameters are set to 256 MB. When the data processing capabilities of services are insufficient, data accumulates in the memory of sidecars, resulting in an increase in memory usage.

Solutions

Configurations stored in sidecars

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management. On the Mesh Management page, find the ASM instance that you want to manage, and click Manage in the Actions column. In the left-side navigation pane of the page that appears, choose Mesh Optimization Center > Sidecar recommendation. On the page that appears, configure sidecar recommendation to determine the services whose configurations will be stored in sidecars. This helps to reduce memory usage of sidecars. For more information, see Configuration push optimization overview.

Traffic of real-time requests

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Sidecar Management(Data Plane) > Sidecar Proxy Setting.
  3. On the Sidecar Proxy Setting page, click the global tab. Then, click enable/disable Sidecar proxy by port or address and configure the block scope.
    For example, you can configure not to block requests from middleware such as Redis (port 2379) or MySQL (port 3306), or requests from outside clusters. Global settings

Metrics with too many statistical values

You can set metrics based on your business requirements. For more information, see Customize metrics in ASM.

gRPC throttling-related data

gRPC requests are related to the sizes of streams corresponding to upstream and downstream services and the size of a single stream buffer. You can log on to the ASM console and set the initial_stream_window_size and initial_connection_window_size parameters to optimize the memory usage of sidecars.

  1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
  2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Plugin Extension Center > Market Place.
  3. On the Market Place page, click Set the initial window size for http2/gprc to set the related parameters.