:Replicate data across clusters with CCR

Solution comparison

For remote disaster recovery scenarios that require a low RPO and low latency, CCR is the best choice:

• CCR synchronizes data in seconds, minimizing data loss.

• If the leader cluster fails, you can redirect traffic to the follower cluster to restore service without waiting for a snapshot to be restored.

• Although the initial deployment cost is higher, CCR is more cost-effective in the long run by preventing business losses caused by data loss.

Architecture

CCR uses an active-passive architecture. The leader cluster receives all write operations, while the follower cluster is read-only and replicates data from the leader cluster.

• Leader cluster: The source cluster that handles all write operations.

• Follower cluster: The destination cluster that is read-only and synchronizes data from the leader cluster by using CCR.

Data replication process

The data replication process in CCR consists of two phases:

Initialization

The follower cluster sends an initialization request to the leader cluster. The leader cluster then transfers all Lucene segment files from the leader index to the follower index, similar to a snapshot restore mechanism.

Incremental synchronization

Each shard in the follower index periodically sends a pull request to the leader cluster to fetch the latest operations since the last synchronization point. By default, this occurs every second. The process is as follows:

1. Locate the pull starting point: The follower cluster maintains a local remote_checkpoint, which indicates the latest operation successfully applied locally. This checkpoint corresponds to the global_checkpoint in the transaction log (translog) of the leader cluster.

2. Read operations from the leader translog: The leader cluster uses the from_seq_no provided by the follower cluster to find the starting position in its translog. It then reads all subsequent operations (index, update, delete) and returns them as a list.

3. Replay operations on the follower cluster: The follower cluster replays these operations locally in order and updates its remote_checkpoint. If a replay fails, for example, due to a version conflict, synchronization is paused, and an error is logged.

4. Continuous polling: The follower cluster continuously polls for new operations at a fixed interval, typically achieving sub-second latency.

Role of the translog

The transaction log (translog) is the data source for incremental synchronization in CCR. It serves the following purposes in Elasticsearch:

• Prevent data loss: By default, Elasticsearch performs a refresh every second, which creates a new searchable Lucene segment from the in-memory buffer. However, this segment is not yet flushed to disk. The translog records all write operations. If a node crashes, you can recover data by replaying the log.

• Ensure replica consistency: Elasticsearch first writes operations to the translog and then forwards them to replica shards. It returns a success response only after both the primary and replica shards confirm the write.

• Support incremental synchronization in CCR: CCR uses an internal Elasticsearch API to read the translog and fetch all changes after a specified sequence number, which enables near real-time data replication.

The translog is stored separately for each shard. Each shard has its own translog directory located at indices/{index_uuid}/{shard_id}/translog/. Translog files (.tlog) are stored in a binary format and managed using a generation mechanism. Elasticsearch creates a new generation file each time a flush occurs or when the file reaches its size limit, which is 512 MB by default.