Switch from Kafka MirrorMaker to Confluent Replicator with Offset Sync - ApsaraMQ for Kafka

Apache Kafka MirrorMaker replicates data between Kafka clusters by reading from topics in a source cluster and writing to identically named topics in a destination cluster. Confluent Replicator provides additional capabilities: it replicates both topic configurations and data, integrates with Kafka Connect for scalability, and supports monitoring through Confluent Control Center.

This guide walks you through replacing a running MirrorMaker instance with Confluent Replicator. Replicator picks up from the last offset that MirrorMaker committed, so legacy messages remain untouched.

Scenario overview

Both scenarios in this guide use two data centers, DC1 and DC2, each running its own Kafka cluster. A single topic named inventory is replicated from DC1 to DC2. The only difference between the two procedures is how partitions are handled:

Scenario	Partition count	Key setting
Same partition count	DC1 = DC2	`topic.preserve.partitions=true`
Different partition count	DC1 ≠ DC2	`topic.preserve.partitions=false`

Prerequisites

These prerequisites apply to both scenarios:

Confluent Platform 5.0.0 or later. For installation instructions, see Install Confluent Platform on-premises.
The src.consumer.group.id value in Confluent Replicator must match the group.id value of the MirrorMaker instance you are replacing. This match makes sure Replicator resumes from the correct committed offset.

Migrate with the same partition count

Use this procedure when the inventory topic has the same number of partitions in both DC1 and DC2.

Additional prerequisite: The partition count for the inventory topic in DC1 must equal the partition count in DC2.

Step 1: Stop MirrorMaker

Stop the running MirrorMaker instance in DC1. Replace <mm-pid> with the MirrorMaker process ID.

kill <mm-pid>

Step 2: Configure the Replicator consumer

Add the following lines to CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_consumer.properties. Replace localhost:9082 with the bootstrap.servers address of your source cluster.

bootstrap.servers=localhost:9082
topic.preserve.partitions=true

Setting topic.preserve.partitions=true makes sure each message is written to the same partition number in DC2 as in DC1.

Step 3: Configure the Replicator producer

Add the following line to CONFLUENT_HOME/etc/kafka-connect-replicator/replicator_producer.properties. Replace localhost:9092 with the bootstrap.servers address of your destination cluster.

bootstrap.servers=localhost:9092

Step 4: Set replication factors

Append replication factor settings to ./etc/kafka-connect-replicator/quickstart-replicator.properties. Set the value to 2 or 3 based on the number of brokers in your cluster.

echo "confluent.topic.replication.factor=3" >> ./etc/kafka-connect-replicator/quickstart-replicator.properties
echo "offset.storage.replication.factor=3" >> ./etc/kafka-connect-replicator/quickstart-replicator.properties
echo "config.storage.replication.factor=3" >> ./etc/kafka-connect-replicator/quickstart-replicator.properties
echo "status.storage.replication.factor=3" >> ./etc/kafka-connect-replicator/quickstart-replicator.properties

Step 5: Start Confluent Replicator

Run Confluent Replicator as a standalone executable or from a Docker image. Replace <new-cluster-id> with a unique identifier for this Replicator instance.

replicator --cluster.id <new-cluster-id> \
  --producer.config replicator_producer.properties \
  --consumer.config replicator_consumer.properties \
  --replication.config ./etc/kafka-connect-replicator/quickstart-replicator.properties

Replicator reads the offsets previously committed by MirrorMaker in DC1 and begins replicating from that point.

Verify the migration

After Replicator starts, confirm that replication is working:

Check Replicator logs for successful initialization and the starting offset.
Consume a few messages from the inventory topic in DC2 and verify they match the expected data from DC1.
Compare the lag between the source and destination topics to confirm Replicator is keeping up.

Migrate with a different partition count

Use this procedure when the inventory topic in DC2 has a different number of partitions than in DC1. Because partition assignments cannot be preserved across different partition counts, Replicator redistributes messages using the default partitioner.