This topic describes the issue of a continuous increase in the memory usage of sidecars, and provides possible causes and solutions.
Issue description
During the memory monitoring for pods, it is found that the memory usage of sidecars keeps increasing.
Causes
The following table describes the possible causes of a continuous increase in the memory usage of sidecars.
Cause | Description |
Configurations stored in sidecars | By default, sidecars store the configurations of all services in clusters, such as endpoint information. The configurations occupy a certain amount of memory space. With more and more services deployed in clusters, the service configurations increase and therefore occupy more memory space. |
Traffic of real-time requests | As a forwarding proxy, Envoy has buffers for storing received messages and response messages and caches queued requests. The buffers and cache occupy a certain amount of memory space. |
Metrics with too many statistical values | Metrics with too many statistical values occupy a large amount of memory space. |
gRPC throttling-related data | The HTTP/2 codec of sidecars has the initial_stream_window_size and initial_connection_window_size parameters. The initial_stream_window_size parameter indicates the initial limit on the number of bytes that may be buffered by a single stream in the HTTP/2 codec buffers. The initial_connection_window_size parameter indicates the initial limit on the number of bytes that may be buffered by the connection across all streams in the HTTP/2 codec buffers. By default, the two parameters are set to 256 MB. When the data processing capabilities of services are insufficient, data accumulates in the memory of sidecars, resulting in an increase in memory usage. |
Solutions
Configurations stored in sidecars
Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management. On the Mesh Management page, find the ASM instance that you want to manage, and click Manage in the Actions column. In the left-side navigation pane of the page that appears, choose Mesh Optimization Center > Sidecar recommendation. On the page that appears, configure sidecar recommendation to determine the services whose configurations will be stored in sidecars. This helps to reduce memory usage of sidecars. For more information, see Configuration push optimization overview.
Traffic of real-time requests
Log on to the ASM console. In the left-side navigation pane, choose .
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose .
On the Sidecar Proxy Setting page, click the global tab. Then, click enable/disable Sidecar proxy by port or address and configure the block scope.
For example, you can configure not to block requests from middleware such as Redis (port 2379) or MySQL (port 3306), or requests from outside clusters.
Metrics with too many statistical values
You can set metrics based on your business requirements. For more information, see Create custom metrics in ASM.
gRPC throttling-related data
gRPC requests are related to the sizes of streams corresponding to upstream and downstream services and the size of a single stream buffer. You can log on to the ASM console and set the initial_stream_window_size and initial_connection_window_size parameters to optimize the memory usage of sidecars.
Log on to the ASM console. In the left-side navigation pane, choose .
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose .
On the Market Place page, click Set the initial window size for http2/grpc to set the related parameters.