E-MapReduce:Migrate data from an EMR Hbase cluster online

Solution introduction

The Snapshot-based approach only supports offline migration from HBase, and cannot keep the source cluster online. Since the source cluster continues to run during migration, the following errors may occur:

• Incremental data loss: Write or update operations generated in the source cluster may not be synchronized to the target cluster during Snapshot export and restoration.

• Data consistency problem: Relying on only Snapshot cannot handle real-time incremental data during migration.

The following solutions can solve these problems:

1. Establish replication relationship (Peer):

   • Use HBase Replication to establish table-level replication relationships between source and target clusters. Disable automatic synchronization during the replication.

   • The source cluster records real-time write and update operations for tables but does not immediately synchronize them to the target cluster.

2. Migrate historical data:

   • Use HBase Snapshot to export historical data from the source cluster.

   • If the storage systems (such as HDFS or OSS-HDFS) of the source and target clusters are interconnected, export the Snapshot to the target cluster. Otherwise, export it to an intermediate path in the source cluster first, then synchronize it to the target cluster.

3. Synchronize incremental data: After historical data restoration is complete, enable the automatic synchronization feature of the Replication Peer. It will replay incremental data generated during migration, ensuring data consistency between source and target.

Through these steps, this solution completely migrates both historical and incremental data while ensuring continuous operation of the source cluster, avoiding data loss.

Precautions

• The compatibility of HBase version :

  • HBase 1.x version: Set hbase.replication=true and restart the primary and backup clusters to enable the Replication feature.

  • HBase 2.x version: The hbase.replication parameter has been removed by default. No additional settings are required.

• Replication configuration: HBase configures Replication on a column family basis. If you need to migrate data for specific column families, ensure they have the Replication property correctly configured.

• Network connectivity:

  • Ensure network connectivity between the source HBase cluster and the target HBase cluster through CEN, leased line, or VPN.

  • Establish network connectivity between the source and target clusters in advance, ensuring the following ports are open to the source cluster:

    • Zookeeper service port: Port 2181 of the ECS instance in the target cluster of Zookeeper.

    • HBase service ports: Ports 16010, 16020, and 16030 of the ECS instance where the target cluster of EMR HBase.

Procedure

Step 2: Create a Snapshot

Run the following command in the HBase Shell of the source cluster to create a Snapshot. It will export historical data for the table to be migrated.

snapshot '${table_name}', '${snapshot_name}'

• Parameters:

  • ${table_name}: The name of the table to be migrated. In this example, it is t1.

  • ${snapshot_name}: The custom Snapshot name. In this example, it is t1-snapshot.

• Example:

  snapshot 't1', 't1-snapshot'

Step 3: Export the Snapshot to the target cluster

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot ${snapshot_name} -copy-to ${target_rootdir_path}

Step 4: Restore historical data using Snapshot

1. Log on to the master node of the target cluster. For more information, see Log on to a cluster.

2. Run the following command to enter HBase Shell.

   hbase shell

3. Run the following commands to restore the Snapshot and enable the table in the target cluster.

   restore_snapshot '${snapshot_name}'
   enable '${table_name}'

   • Parameters:

     • ${snapshot_name}: The custom Snapshot name.

     • ${table_name}: The name of the table to be migrated.

   • Examples:

     restore_snapshot 't1-snapshot'
     enable 't1'

Step 5: Enable incremental data synchronization

Run the following command in the HBase Shell of the source cluster to enable Peer synchronization.

enable_peer '${peer_name}'

Example:

enable_peer 'peer1'

Step 6: Verify data

Verify whether the data is complete after migration.

• Small data volume: Use Scan for verification.

  scan '${table_name}'

  Example:

  scan 't1'

• Medium data volume: Use count for verification.

  count '${table_name}'

• Big data volume: Use get for sample verification.

  get '${table_name}', '${rowkey}'

  The ${rowkey} in the code is the unique identifier for each row in the HBase table.

Step 7: Delete the Snapshot

Run the following command to delete all snapshots related to the target table after the verification to free up storage space.

delete_table_snapshots '${table_name}'

Example:

delete_table_snapshots 't1'

Step 8: Clean up the Peer

After incremental data synchronization is complete, you need to perform dual-write or cutover operations for the application to ensure that all read and write requests are switched to the target HBase cluster. To avoid duplicate data synchronization, you need to delete the replication relationship (Peer) between the source and target clusters after the switch.

1. Migrate the client.

   Switch the upstream and downstream applications involving the HBase cluster to the target HBase cluster, including applications that read and write HBase through API or command line. To do this, perform the following operations:

   • Update connection configuration: Modify the application's configuration files or code to switch the HBase connection information (such as Zookeeper address, port, etc.) from the source cluster to the target cluster.

   • Verify functionality: Ensure that the application can normally read and write to the target cluster. Run necessary functional testing and data validation.

   • Dual-run or cutover: Based on business requirements, choose to dual-run (read and write to both source and target clusters simultaneously) or directly cut over to the target cluster.

2. Disable the automatic synchronization feature of the Peer.

   Disable the automatic synchronization feature of the specified Peer in the HBase Shell of the source cluster. Ensure that data synchronization stops immediately.

   disable_peer '${peer_name}'

   Example:

   disable_peer 'peer1'

3. Delete the Peer.

   After disabling the Peer, delete the Peer to disconnect the replication relationship completely between the source and target clusters.

   remove_peer '${peer_name}'

   Example:

   remove_peer 'peer1'

4. Verify whether the Peer has been successfully deleted.

   Run the following command to list all current Peers to ensure that peer1 in this example has been deleted.

   list_peers

FAQ