Tair (Redis® OSS-Compatible):Best practices for the connection pool feature in cluster proxy mode

Scenarios

In the proxy mode of a cloud native cluster instance, standard commands are forwarded through shared connections. An increase in client connections does not affect the number of backend DB connections. When your business logic includes specific stateful commands, the connection between a proxy and a data node switches from shared mode to exclusive mode.

In exclusive mode, each client connection establishes a corresponding exclusive connection on the data node. In high-concurrency scenarios, this causes the number of connections on the data node to grow linearly with the number of client connections. This can trigger a max number of clients reached error and lead to a service interruption.

When you use watch or blocking requests, scaling out by increasing the number of shards does not resolve the issue of too many connections on a single shard.

The commands that trigger exclusive mode include the following:

• Transaction commands: MULTI/EXEC (after WATCH)

• Blocking commands: BLPOP, BRPOP, BRPOPLPUSH, XREAD ... BLOCK ...

The connection pool feature pools and reuses exclusive connections from proxies to data nodes. This reduces the number of connections on data nodes and prevents request failures caused by exceeding the connection limit.

Architecture

The connection pool feature runs independently on each proxy child instance. Each proxy maintains a dedicated connection pool for its connections to each data node.

Example:

Without a connection pool, each connection between a client and a proxy establishes a separate connection to each backend data shard it accesses. For example, if 10 client connections all access 5 shards, a total of 10 × 5 = 50 connections are created.

When you enable the connection pool, the proxy maintains a limited connection pool for each data shard. Client requests then reuse the connections in these pools. This significantly reduces the number of database connections that are actually established. For example, for the same 10 client connections, each shard might only need to maintain 2 connections in its pool, resulting in a total of only 2 × 5 = 10 connections.

Workflow:

1. When a client connection (Client Conn) initiates a request in exclusive mode, the proxy node retrieves an idle connection from the corresponding data shard's connection pool.

2. Pool hit: If an idle connection is available in the pool, the proxy immediately attaches the connection to the client connection to forward the request. After the request is complete, the connection is detached and returned to the connection pool for other clients to reuse.

3. Pool miss: If no idle connections are available in the pool, the proxy creates a new connection. After the request is complete:

   1. If the total number of connections in the pool has not reached the upper limit (private_conn_pool_max_idle_per_db), the connection is added to the pool.

   2. If the total number of connections in the pool has reached the upper limit, the connection is immediately released (short-lived connection mode).

This optimization greatly reduces connection management overhead on the database, decreases resource overhead, and improves database stability and throughput.

Applicability

The instance must meet the following requirements:

• Deployment mode: Cloud native. You can change the deployment mode from classic to cloud native.

• Instance architecture: Cluster. You can change the instance architecture from standard to cluster.

• Connection mode: Proxy mode.

• Proxy version: 7.0.17 or later. If your version is earlier, you can upgrade the proxy version.

Enable the connection pool feature

You can enable the connection pool feature by modifying an instance parameter. This operation supports hot updates and does not require an instance restart. It does not affect your services.

1. Go to the Instances page. In the top navigation bar, select a region. Then, click the ID of the target instance.

2. In the navigation pane on the left, click Parameter Settings.

3. In the parameter list, find private_conn_pool_enabled and click Modify in the Actions column.

4. In the dialog box that appears, set the parameter value to 1 and click OK.

Afterward, you can go to the Cloud Monitor console, find the instance, and view the Connections from Proxy to DB metric under Infrastructure Monitoring to compare the data before and after you enable the feature.

Configuration details and tuning guide

Notes

• Set a reasonable value for private_conn_pool_min_idle_per_db

  Each idle connection consumes some memory resources on the backend DB node. In instances with multiple shards and multiple proxy nodes, setting private_conn_pool_min_idle_per_db to a high value can lead to high memory usage.

• Avoid unnecessary DB context switching

  When a connection from the pool is reused by a client that accesses a different DB index, the proxy must run the SELECT command to switch the context. This operation increases the DB payload and request latency. In your application design, ensure that all clients use the same DB index, which is typically DB 0.

• Not applicable to publish/subscribe commands

  The connection pool feature does not apply to publish/subscribe commands such as SUBSCRIBE and PSUBSCRIBE. These commands permanently occupy a connection, so these connections are not managed by the pool.

References

• Cluster architecture

• Tair proxy features