Use Raft-RocksDB-Tablestore as the backend storage for Jindo metadata - E-MapReduce

E-MapReduce (EMR) 3.27.0 and later allows you to use Raft-RocksDB-Tablestore as the storage backend for the JindoFS Namespace Service. Three master nodes form a Raft instance, and each peer node stores metadata in a local RocksDB.

Prerequisites

Create a Tablestore instance. A high-performance instance is recommended. For more information, see Activate Tablestore and create an instance.

Note
You must enable the transaction feature.
Create an E-MapReduce cluster with three master nodes. For more information, see Create a cluster. When you create the cluster, enable High Availability and select 3 Master for the Deployment Mode.

Background information

RocksDB uses the Raft protocol to replicate data across three master nodes. You can bind a cluster to a Tablestore instance as an additional storage backend, and local metadata is asynchronously synchronized to that instance.

The following figure shows the high-availability architecture of the Namespace Service with Raft, RocksDB, and Tablestore. Raft + RocksDB + Tablestore

Configure the local Raft backend

After you create the EMR cluster, stop all SmartData services.
1. Log on to the Alibaba Cloud EMR console.
2. In the top navigation bar, select a region and resource group as required.
3. Click the Clusters tab.
4. On the Clusters page, find the target cluster and click Details in the Actions column.
5. In the navigation pane on the left, choose Services > SmartData.
6. In the upper-right corner, choose Actions > > Stop All Components.
Add namespaces as required.
Go to the bigboot tab of the SmartData service.
1. In the navigation pane on the left, choose Services > SmartData.
2. Click the Configure tab.
3. In the Service Configuration section, click the bigboot tab.

On the bigboot tab of the SmartData service, set the following parameters.

Parameter	Description	Example
namespace.backend.type	Storage backend for the Namespace Service. Valid values: `rocksdb` `ots` `raft` The default value is `rocksdb`.	`raft`
namespace.backend.raft.initial-conf	Addresses of the three master nodes in the Raft instance. This value is fixed.	`emr-header-1:8103:0,emr-header-2:8103:0,emr-header-3:8103:0`
jfs.namespace.server.rpc-address	Client access addresses for the three Raft master nodes. This value is fixed.	`emr-header-1:8101,emr-header-2:8101,emr-header-3:8101`

Note

To use Tablestore as a remote storage backend, perform step 5 through step 7. Otherwise, skip to step 6 and step 7.

Optional: Configure Tablestore for remote asynchronous storage.

On the bigboot tab of the SmartData service, set the following parameters.

Parameter	Description	Example
namespace.ots.instance	Name of the Tablestore instance.	`emr-jfs`
namespace.ots.accessKey	AccessKey ID for the Tablestore instance.	YourAccessKeyID
namespace.ots.accessSecret	AccessKey secret for the Tablestore instance.	YourAccessKeySecret
namespace.ots.endpoint	Endpoint of the Tablestore instance. We recommend that you use a VPC endpoint for EMR clusters.	`http://emr-jfs.cn-hangzhou.vpc.tablestore.aliyuncs.com`
namespace.backend.raft.async.ots.enabled	Whether to enable asynchronous data upload to Tablestore. Valid values: `true` `false` If you set this parameter to `true`, you must enable this feature before the SmartData service is initialized. Note You cannot enable this feature after the service is initialized because the data in Tablestore would be older than the data in the local RocksDB.	`true`

Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the change and turn on Auto-update Configuration.
3. Click OK.
In the upper-right corner, choose Actions > > Start All Components.

Restore metadata from Tablestore

If you enabled remote asynchronous storage with Tablestore on the original cluster, the Tablestore instance contains a complete copy of JindoFS metadata. After you stop or release the original cluster, you can restore the metadata to a new cluster to resume file access.

Optional: Perform preparatory work.

Optional: Collect metadata statistics (file and folder counts) from the original cluster.

[hadoop@emr-header-1 ~]$ hadoop fs -count jfs://test/
        1596      1482809                 25 jfs://test/
    (Number of folders) (Number of files)

Stop all jobs on the original cluster and wait for 30 to 120 seconds for the metadata to fully synchronize to Tablestore. Run the following command to check the status. If the LEADER node shows _synced=1, the data in Tablestore is fully synchronized.

jindo jfs -metaStatus -detail

The following is a sample output. state: LEADER indicates that the current Raft node is a leader. The log index has been updated to 624625, and all replicas are synchronized.

[RaftPeerImpl]
peer_id: xxx
state: LEADER
readonly: 0
term: 2
conf_index: 1
peers: xxx
changing_conf: NO    stage: STAGE_NONE
election_timer: timeout(5000ms) STOPPED
vote_timer: timeout(5000ms) STOPPED
stepdown_timer: timeout(5000ms) SCHEDULING(in 2335ms)
snapshot_timer: timeout(3600000ms) SCHEDULING(in 150305ms)
storage: [1, 624625]
disk_index: 624625
known_applied_index: 624625
last_log_id: (index=624625,term=2)
first_index_pinned: 624625
state_machine: Idle
last_committed_index: 624625
last_snapshot_index: 0
last_snapshot_term: 0
snapshot_status: IDLE
replicator_25769803789@xxx  next_index=624626   flying_append_entries_size=0 idle hc=2301 ac=624261 ic=0
replicator_32985348833259@xxx    next_index=624626   flying_append_entries_size=0 idle hc=2301 ac=623564 ic=0
OtsUploader: _lastStopIndex=624624, _synced=1

Stop or release the original cluster to ensure that no other clusters are accessing the Tablestore instance.

Create a new cluster.

Create a new EMR cluster in the same region as the Tablestore instance, and then stop all SmartData services. For more information, see Step 1 in Configure the local Raft backend.

Initialize the configuration.

On the bigboot tab of the SmartData service, set the following parameters.

Parameter

Description

Example

namespace.backend.raft.async.ots.enabled

Whether to enable asynchronous data upload to Tablestore. Valid values:

true
false

false

namespace.backend.raft.recovery.mode

Whether to enable metadata recovery from Tablestore. Valid values:

true
false

true

Save the configuration.
1. In the upper-right corner, click Save.
2. In the Confirm dialog box, enter a reason for the change and turn on Auto-update Configuration.
3. Click OK.
In the upper-right corner, choose Actions > > Start All Components.

After the SmartData service on the new cluster starts, metadata is automatically restored from Tablestore to the local Raft-RocksDB. You can run the following command to monitor the recovery progress.

jindo jfs -metaStatus -detail

After you run the command, confirm that the state in the output is LEADER and the state in the [Recovery From OTS Status] section is FINISH. This indicates that recovery is complete. The following is a sample output:

[RaftPeerImpl]
peer_id: xxx:8103:0
state: LEADER
readonly: 0
term: 2
conf_index: 1
peers: xxx
changing_conf: NO    stage: STAGE_NONE
election_timer: timeout(5000ms) STOPPED
vote_timer: timeout(5000ms) STOPPED
stepdown_timer: timeout(5000ms) SCHEDULING(in 3382ms)
snapshot_timer: timeout(600000ms) SCHEDULING(in 474855ms)
storage: [1, 153]
disk_index: 153
known_applied_index: 153
last_log_id: (index=153,term=2)
first_index_pinned: 1
state_machine: Idle
last_committed_index: 153
last_snapshot_index: 1
last_snapshot_term: 2
snapshot_status: IDLE
replicator_1116691496965@xxx: next_index=154   flying_append_entries_size=0 idle hc=262 ac=154 ic=0
replicator_3311419785217@xxx: next_index=154   flying_append_entries_size=0 idle hc=262 ac=154 ic=0

[Recovery From OTS Status]
state: FINISH
[Recovery From OTS Status]
state: FINISH
total rows: 1484409
table `jfs_block_test` 2 rows.
table `jfs_namespace_cache_ns` 1 rows.
table `jfs_namespace_test` 1484406 rows.

Optional: Verify that the number of files on the new cluster is consistent with that on the original cluster.

The cluster is now in read-only recovery mode.

# Compare the file count to verify that it is consistent with that of the original cluster.
[hadoop@emr-header-1 ~]$ hadoop fs -count jfs://test/
        1596      1482809                 25 jfs://test/
# Files can be read normally by using the cat or get command.
[hadoop@emr-header-1 ~]$ hadoop fs -cat jfs://test/testfile
this is a test file
# View the directory contents.
[hadoop@emr-header-1 ~]$ hadoop fs -ls jfs://test/
Found 3 items
drwxrwxr-x   - root   root            0 2020-03-25 14:54 jfs://test/emr-header-1.cluster-50087
-rw-r-----   1 hadoop hadoop          5 2020-03-25 14:50 jfs://test/haha-12096RANDOM.txt
-rw-r-----   1 hadoop hadoop         20 2020-03-25 15:07 jfs://test/testfile
# Files cannot be modified because the cluster is in read-only mode.
[hadoop@emr-header-1 ~]$ hadoop fs -rm jfs://test/testfile
java.io.IOException: ErrorCode : 25021 , ErrorMsg: Namespace is under recovery mode, and is read-only.

Update the configuration to switch the cluster to normal mode and enable asynchronous data upload to Tablestore.

On the bigboot tab of the SmartData service, set the following parameters.

Parameter

Description

Example

namespace.backend.raft.async.ots.enabled

Whether to enable asynchronous data upload to Tablestore. Valid values:

true
false

true

namespace.backend.raft.recovery.mode

Whether to enable metadata recovery from Tablestore. Valid values:

true
false

false

Restart the cluster.
1. Click the Clusters tab.
2. On the Clusters page, find the target cluster and choose More > Restart in the Actions column.