Elasticsearch:Replicate data across clusters with CCR

Basic architecture

CCR uses an active-passive architecture. The leader cluster handles all writes, and the follower cluster replicates data in read-only mode.

• leader cluster: The source cluster that accepts all write operations.

• follower cluster: The destination cluster that is read-only and replicates data from the leader cluster.

Data replication process

CCR data replication consists of two phases:

Initialization phase

The follower cluster requests initialization from the leader, which transfers all Lucene segment files to the follower index, similar to snapshot recovery.

Incremental synchronization phase

By default, follower index shards pull the latest operations from the leader cluster every second:

1. Determine the pull starting point: The follower cluster maintains a local remote_checkpoint that tracks the latest operation applied to the local index, corresponding to the global_checkpoint in the leader's Translog.

2. Read from the leader's Translog: The leader uses the from_seq_no from the follower to locate the starting position in the Translog, reads all subsequent operations (index, update, delete), and returns them.

3. Replay on the follower: The follower replays operations in order and updates its remote_checkpoint. If a replay fails (for example, due to a version conflict), synchronization pauses and logs an error.

4. Continuous polling: The follower polls for new operations at fixed intervals, achieving sub-second latency.

Translog in CCR

The Translog (transaction log) is the data source for CCR incremental synchronization. It serves these purposes:

• Prevent data loss: Records all write operations for replay-based recovery after node crashes.

• Ensure replica consistency: Writes go to the Translog first, then to the replica shard. An operation succeeds only after both primary and replica shards acknowledge it.

• Support CCR incremental synchronization: CCR reads operation logs through the internal Translog API to retrieve all changes after a specified sequence number for near-real-time replication.

Each shard has its own Translog directory at indices/{index_uuid}/{shard_id}/translog/. Translog files (.tlog) use binary format with a generation mechanism. A new generation file is created on each flush or when the file reaches 512 MB (default).