Migrate HBase data online

Solution overview

Traditional snapshot-based HBase migration only supports offline migration, which cannot meet the requirement of keeping the source cluster running. Because the source cluster remains active during the migration, the following issues may occur:

• Incremental data loss: Write or update operations performed on the source cluster during snapshot export and restoration might not be synchronized to the destination cluster.

• Data consistency is difficult to guarantee: Snapshots alone cannot handle the real-time incremental data that is generated during the migration.

To address these issues, you can use the following online migration approach:

1. Establish a replication relationship (peer):

   • Use HBase Replication to create a table-level replication relationship between the source and destination clusters and then temporarily disable automatic synchronization.

   • The source cluster records real-time writes and changes to the table but does not immediately synchronize them to the destination cluster.

2. Migrate historical data:

   • Use HBase Snapshot to export the historical data from the source cluster.

   • If the storage systems, such as HDFS or OSS-HDFS, of the source and destination clusters are interconnected, you can export the snapshot directly to the destination cluster. Otherwise, you must first export it to an intermediate path on the source cluster and then synchronize it to the destination cluster.

3. Synchronize incremental data: After you restore the historical data, enable automatic synchronization for the replication peer to replay the incremental data that was generated during the migration. This ensures data consistency between the source and destination clusters.

These steps allow you to fully migrate both historical and incremental data while the source cluster continues to run, which avoids data loss.

Important considerations

• HBase version compatibility:

  • HBase 1.x: Set hbase.replication=true and restart both the master and standby clusters to enable replication.

  • HBase 2.x: The hbase.replication parameter is removed by default and does not require additional configuration.

• Replication configuration: HBase enables replication for each column family. To migrate specific column families, you must ensure that their replication property is correctly configured.

• Network connectivity:

  • Use Cloud Enterprise Network (CEN), leased lines, or a VPN to ensure network connectivity between the source and destination HBase clusters.

  • Establish network connectivity between the source and destination clusters. Ensure that the following ports are accessible from the source cluster:

    • ZooKeeper service port: Port 2181 on the ECS instance that hosts ZooKeeper in the destination cluster.

    • HBase service ports: Ports 16010, 16020, and 16030 on the ECS instance that hosts EMR HBase in the destination cluster.

Procedure

Step 2: Create a Snapshot

In the HBase Shell of the source cluster, run the following command to create a snapshot of the table to migrate. This snapshot is used to export historical data.

snapshot '${table_name}', '${snapshot_name}'

• Parameters:

  • ${table_name}: The name of the table to migrate. This topic uses t1 as an example.

  • ${snapshot_name}: A custom snapshot name. This topic uses t1-snapshot as an example.

• Example:

  snapshot 't1', 't1-snapshot'

Step 3: Export the Snapshot to the destination cluster

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot ${snapshot_name} -copy-to ${target_rootdir_path}

Step 4: Restore historical data using the Snapshot

1. Log on to the master node of the destination cluster. For more information, see Log on to a cluster.

2. Run the following command to enter the HBase Shell.

   hbase shell

3. Run the following commands to restore the snapshot and enable the table on the destination cluster.

   restore_snapshot '${snapshot_name}'
   enable '${table_name}'

   • Parameters:

     • ${snapshot_name}: A custom snapshot name.

     • ${table_name}: The name of the table to migrate.

   • Example:

     restore_snapshot 't1-snapshot'
     enable 't1'

Step 5: Enable incremental data synchronization

In the HBase Shell of the source cluster, run the following command to enable peer synchronization.

enable_peer '${peer_name}'

Example:

enable_peer 'peer1'

Step 6: Validate data

Verify that the migrated data is complete.

• Small data volume: Use the scan command for validation.

  scan '${table_name}'

  Example:

  scan 't1'

• Medium data volume: Use the count command for validation.

  count '${table_name}'

• Large data volume: Use the get command for sampling validation.

  get '${table_name}', '${rowkey}'

  In the code, ${rowkey} is the unique identifier for each row in the HBase table.

Step 7: Delete the Snapshot

After the validation is successful, run the following command to delete all snapshots that are related to the destination table and free up storage space.

delete_table_snapshots '${table_name}'

Example:

delete_table_snapshots 't1'

Step 8: Clean up the Peer

After the incremental data synchronization is complete, perform dual-write or cutover operations to switch all read and write requests to the destination HBase cluster. To prevent duplicate data synchronization, delete the replication relationship (peer) between the source and destination clusters after the switch is complete.

1. Migrate clients.

   Switch the upstream and downstream applications that interact with the HBase cluster to the destination HBase cluster. This includes applications that read data from or write data to HBase through APIs or the command line. Perform the following steps:

   • Update the connection configuration: Modify the application configuration files or code to change the HBase connection information, such as the ZooKeeper address and port, from the source cluster to the destination cluster.

   • Validate functionality: Ensure that the application can read data from and write data to the destination cluster as expected. You must also perform the necessary functional testing and data validation.

   • Dual-run or cutover: Based on your business needs, you can choose to perform a dual-run, where you read data from and write data to both the source and destination clusters simultaneously, or perform a direct cutover to the destination cluster.

2. Disable automatic synchronization for the peer.

   In the HBase Shell of the source cluster, disable automatic synchronization for the specified peer to immediately stop data synchronization.

   disable_peer '${peer_name}'

   Example:

   disable_peer 'peer1'

3. Delete the peer.

   After you disable the peer, delete it to completely remove the replication relationship between the source and destination clusters.

   remove_peer '${peer_name}'

   Example:

   remove_peer 'peer1'

4. Verify that the peer was deleted.

   Run the following command to list all current peers and confirm that peer1 in this example is deleted.

   list_peers

FAQ