Tair imperceptible switching - Tair (Redis® OSS-Compatible)

Background

In open-source databases such as Redis or Valkey, the mainstream high availability architectures are:

Standalone: The primary node provides read and write services, and the secondary node serves as a failover candidate. The community typically introduces Sentinel as an independent monitoring and arbitration component that automatically initiates failover when the primary node becomes unreachable.
Cluster: The entire keyspace is divided into 16,384 slots distributed across multiple master nodes, each with optional replica nodes. When a master fails, its replica is automatically promoted through a voting mechanism. Clients achieve automatic routing failover upon receiving MOVED/ASK redirects.

In these architectures, version upgrades, HA failovers, or scaling operations inevitably cause connection interruptions, errors, or data migration blocking. Tair provides imperceptible switching capabilities for both cluster and standalone architectures.

Cluster architecture seamless scaling

During scaling (adding or removing shards), Tair uses Slot replication-based seamless data migration to eliminate the -ASK and -TRYAGAIN errors that native Redis clusters may produce.

Open-source Redis Cluster data migration: principles and pain points

Open-source Redis Cluster migrates Slots using the Per Key approach, completing a single Slot migration by traversing and migrating keys in batches. The specific flow is as follows:

Set the importing state on the destination shard.
Set the migrating state on the source shard.
Retrieve the key list of the Slot to be migrated from the source shard.
Send the migrate command to the source shard with a batch of keys. The migrate command is synchronous and blocking. It sends keys to the destination shard for restore, then deletes them from the source. This repeats until the key list is empty.
Use the setslot command to assign the migrated Slot to the destination node.

The Per Key migration approach has the following pain points:

The migrate command is synchronous and blocking, preventing other requests from being processed. Migrating large keys causes prolonged blocking.
When a key does not exist on the source shard, the request is redirected to the destination shard, adding an extra routing hop.
Multi-key command requests may receive a TRYAGAIN error.
Data of a single Slot is spread across two shards. If an exception occurs, the migration cannot be rolled back smoothly.

Tair Slot replication-based seamless data migration

Building on primary-replica replication, Tair developed Slot replication technology (Slot Mig) at the kernel level. It synchronizes data between nodes by replicating Slots, combined with a central controller (CS) that controls millisecond-level Slot Wait write-blocking. When the cluster topology is switched, client requests are redirected via MOVED semantics.

The specific flow of Slot Mig migration is as follows:

CS sends the slotRepl migration command to the destination shard.
The destination node sends a slotPsync command to the source node.
The source node generates a snapshot of the Slot to be migrated.
The destination node receives the snapshot data.
The source node synchronizes incremental data to the destination node.
CS dynamically evaluates the sync gap and sends slotReplWait to block writes at the millisecond level. If the blocking exceeds the threshold, a rollback is automatically triggered.
CS switches the cluster topology after sync completes. Old client requests are redirected via MOVED semantics.
The source node deletes the migrated Slot data.

Slot Mig migration has the following advantages:

No read-only errors and no connection drops.
MOVED semantics only after topology switch, with no ASK semantics.
Large keys and multi-key requests are not affected.
Data of a single Slot is no longer split, and the process can be rolled back in one step.

Comparison: Tair Slot Mig vs. community Per Key

Item	Tair Slot Mig	Community Per Key
Request status during migration	Two states: normal access before switch; MOVED redirect after, no ASK.	Three states: MOVED/ASK redirect; TRYAGAIN error; blocked during migrate.
Read-only errors	No	No
Write blocking	Millisecond-level Slot blocking.	Multi-key with missing keys; migrate command blocking.
Connection drops	No	No
Rollback before completion	Yes	No

Notes

For direct connection mode, the client must handle MOVED commands.
For proxy mode, scaling down releases proxy nodes, causing connection drops. The client must have reconnection capability.
Non-data commands (such as PING, INFO) and PUB/SUB and BLOCKING commands cannot be automatically redirected after Slot migration. The application must periodically update the routing table.
To ensure data consistency, write request latency for the corresponding Slot may increase during the final stage of migration, but will not fail.

Cluster architecture imperceptible HA switching

According to the Redis standard protocol RESP, after a primary-replica switch in a cluster, the replica returns MOVED to existing requests. The client refreshes its routing table and retries, achieving imperceptible switching.

Tair enhanced the direct-connect cluster architecture with the following improvements:

Introduced VIP mechanism to ensure link connectivity after switching.
Replaced gossip with a centralized Config Server for unified routing table distribution, eliminating distributed protocol uncertainty.

Standalone architecture imperceptible switching

Unlike cluster architecture's native redirection capability, standalone architecture faces two challenges: protocol incompatibility between standalone and cluster modes, and Sentinel's inability to control kernel-level connection behavior. Tair achieved imperceptible switching through three layers of innovation.

Kernel layer: REDIRECT protocol

Tair implemented the imperceptible switching protocol at the kernel level, which was merged into the Valkey community. The new protocol format is REDIRECT HOST PORT, where REDIRECT indicates the client should redirect the request to the target node.

Client layer: SDK support

Tair implemented imperceptible switching support in the Valkey-Java client. For in-flight connections, a two-layer lock mechanism controls connection pool reinitialization: the first layer uses ReentrantLock's tryLock for candidate renewal, and the second uses ReentrantReadWriteLock for read-write control.

Clients must be upgraded to Valkey-Java 5.3.0 or later:

<dependency>
  <groupId>io.valkey</groupId>
  <artifactId>valkey-java</artifactId>
  <version>5.3.0</version>
</dependency>

Network layer: Connection Draining

Tair uses an "LB + VIP" architecture. After VIP switching, the Connection Draining mechanism keeps old connections alive for a configurable period, ensuring the old primary has enough time to deliver the REDIRECT command before gracefully closing the connection.

Test results

Using Tair-pulse (an open-source tool for evaluating client errors and RT during switching), comparative tests show:

The original approach encounters 5 types of errors; imperceptible switching encounters zero errors.
The original approach has ~5 seconds of unavailability; imperceptible switching has ~1 second of increased RT.

Note

The imperceptible switching approach uses pause write, which may cause command timeouts (up to ~1 second). You can increase the timeout to ensure zero errors.

Limitations

Cluster architecture seamless scaling limitations

The instance must use cloud disks.
The instance must use the cluster architecture.
Supported versions: Redis OSS 5.0 (minor 5.2.0+), 6.0 (minor 6.0.2.0+), 7.0; Tair Memory-Optimized compatible with Redis 5.0 (minor 5.0.34+), 6.0, 7.0; Tair Persistent Memory; Tair Disk-Based.
The client must handle MOVED commands (direct mode) or have reconnection capability (proxy mode).

Standalone architecture imperceptible switching limitations

Imperceptible switching for standalone architecture has the following minor version requirements:

Redis OSS: Only version 7.0, minor version 0.2.9 or later.
Tair (Enterprise): All major versions, minor version 25.11.0.0 or later.

Note

Valkey-Java 5.3.0 or later client is also required.