All Products
Search

This Product

    E-MapReduce:Use Raft-RocksDB-Tablestore as the storage backend

    Document Center

    E-MapReduce:Use Raft-RocksDB-Tablestore as the storage backend

    Last Updated:Jun 20, 2026

    JindoFS in E-MapReduce (EMR) 3.27.0 and later supports Raft-RocksDB-Tablestore as the storage backend for its Namespace Service. In an EMR JindoFS cluster, three master nodes form a Raft instance, and each peer node uses a local RocksDB to store metadata.

    Prerequisites

    • Create a Tablestore instance. A high-performance instance is recommended. For more information, see Activate Tablestore and create an instance.

      Note

      You must enable the transaction feature.

    • Create an E-MapReduce cluster with three master nodes. For more information, see Create a cluster. When you create the cluster, enable High Availability and select 3 Master for the Deployment Mode.

    Background information

    RocksDB uses the Raft protocol to replicate data across the three master nodes. You can bind a cluster to a Tablestore instance, which serves as an additional storage backend for the JindoFS Namespace Service. Local metadata is asynchronously synchronized to your Tablestore instance.

    The following figure shows the high-availability architecture that uses Raft, RocksDB, and Tablestore for the Namespace Service.Raft + RocksDB + Tablestore

    Configure the local Raft backend

    1. After you create the EMR cluster, stop all SmartData services.

      1. Log on to the Alibaba Cloud EMR console.

      2. In the top navigation bar, select a region and resource group as required.

      3. Click the Clusters tab.

      4. On the Clusters page, find the target cluster and click Details in the Actions column.

      5. In the navigation pane on the left, choose Services > SmartData.

      6. In the upper-right corner, choose Actions > > Stop All Components.

    2. Add namespaces as required.

    3. Go to the bigboot tab of the SmartData service.

      1. In the navigation pane on the left, choose Services > SmartData.

      2. Click the Configure tab.

      3. In the Service Configuration section, click the bigboot tab.

    4. On the bigboot tab of the SmartData service, set the following parameters.

      Parameter

      Description

      Example

      namespace.backend.type

      Storage backend for the Namespace Service. Valid values:

      • rocksdb

      • ots

      • raft

      The default value is rocksdb.

      raft

      namespace.backend.raft.initial-conf

      Addresses of the three master nodes that host the Raft instance. This value is fixed.

      emr-header-1:8103:0,emr-header-2:8103:0,emr-header-3:8103:0

      jfs.namespace.server.rpc-address

      Client access addresses for the three master nodes of the Raft instance. This value is fixed.

      emr-header-1:8101,emr-header-2:8101,emr-header-3:8101
      Note

      To use Tablestore as a remote storage backend, perform step 5 through step 7. Otherwise, skip to step 6 and step 7.

    5. Optional: Configure Tablestore for remote asynchronous storage.

      On the bigboot tab of the SmartData service, set the following parameters.

      Parameter

      Description

      Example

      namespace.ots.instance

      The name of the Tablestore instance.

      emr-jfs

      namespace.ots.accessKey

      AccessKey ID for the Tablestore instance.

      YourAccessKeyID

      namespace.ots.accessSecret

      AccessKey secret for the Tablestore instance.

      YourAccessKeySecret

      namespace.ots.endpoint

      Endpoint of the Tablestore instance. For EMR clusters, using a VPC endpoint is recommended.

      http://emr-jfs.cn-hangzhou.vpc.tablestore.aliyuncs.com

      namespace.backend.raft.async.ots.enabled

      Specifies whether to enable asynchronous data upload to Tablestore. Valid values:

      • true

      • false

      If you set this parameter to true, you must enable this feature before the SmartData service is initialized.

      Note

      You cannot enable this feature after the service is initialized because the data in Tablestore would be older than the data in the local RocksDB.

      true

    6. Save the configuration.

      1. In the upper-right corner, click Save.

      2. In the Confirm dialog box, enter a reason for the change and turn on Auto-update Configuration.

      3. Click OK.

    7. In the upper-right corner, choose Actions > > Start All Components.

    Restore metadata from Tablestore

    If you enabled remote asynchronous storage using Tablestore on the original cluster, the Tablestore instance contains a complete copy of the JindoFS metadata. After you stop or release the original cluster, you can restore the metadata to a new cluster to resume access to your files.

    1. Optional: Perform preparatory work.

      1. Optional: Collect metadata statistics (file and folder counts) from the original cluster.

        [hadoop@emr-header-1 ~]$ hadoop fs -count jfs://test/
        1596      1482809                 25 jfs://test/
    (Number of folders) (Number of files)

      2. Stop all jobs on the original cluster and wait for 30 to 120 seconds for the metadata to fully synchronize to Tablestore. Run the following command to check the status. If the LEADER node shows _synced=1, the data in Tablestore is fully synchronized.

        jindo jfs -metaStatus -detail

        The following is a sample command output. state: LEADER indicates that the current Raft node is a LEADER node. The log index has been updated to 624625, and all replicas are synchronized.

        [RaftPeerImpl]
peer_id: xxx
state: LEADER
readonly: 0
term: 2
conf_index: 1
peers: xxx
changing_conf: NO    stage: STAGE_NONE
election_timer: timeout(5000ms) STOPPED
vote_timer: timeout(5000ms) STOPPED
stepdown_timer: timeout(5000ms) SCHEDULING(in 2335ms)
snapshot_timer: timeout(3600000ms) SCHEDULING(in 150305ms)
storage: [1, 624625]
disk_index: 624625
known_applied_index: 624625
last_log_id: (index=624625,term=2)
first_index_pinned: 624625
state_machine: Idle
last_committed_index: 624625
last_snapshot_index: 0
last_snapshot_term: 0
snapshot_status: IDLE
replicator_25769803789@xxx  next_index=624626   flying_append_entries_size=0 idle hc=2301 ac=624261 ic=0
replicator_32985348833259@xxx    next_index=624626   flying_append_entries_size=0 idle hc=2301 ac=623564 ic=0
OtsUploader: _lastStopIndex=624624, _synced=1

      3. Stop or release the original cluster to ensure that no other clusters are accessing the Tablestore instance.

    2. Create a new cluster.

      Create a new EMR cluster in the same region as the Tablestore instance, and then stop all SmartData services. For more information, see Step 1 in Configure the local Raft backend.

    3. Initialize the configuration.

      On the bigboot tab of the SmartData service, set the following parameters.

      Parameter

      Description

      Example

      namespace.backend.raft.async.ots.enabled

      Specifies whether to enable asynchronous data upload to Tablestore. Valid values:

      • true

      • false

      false

      namespace.backend.raft.recovery.mode

      Specifies whether to enable metadata recovery from Tablestore. Valid values:

      • true

      • false

      true

    4. Save the configuration.

      1. In the upper-right corner, click Save.

      2. In the Confirm dialog box, enter a reason for the change and turn on Auto-update Configuration.

      3. Click OK.

    5. In the upper-right corner, choose Actions > > Start All Components.

    6. After the SmartData service on the new cluster starts, metadata is automatically restored from Tablestore to the local Raft-RocksDB. You can run the following command to monitor the recovery progress.

      jindo jfs -metaStatus -detail

      After you run the command, confirm that the state in the output is LEADER and that the state in the [Recovery From OTS Status] section is FINISH. This indicates the recovery is complete. The following is a sample output:

      [RaftPeerImpl]
peer_id: xxx:8103:0
state: LEADER
readonly: 0
term: 2
conf_index: 1
peers: xxx
changing_conf: NO    stage: STAGE_NONE
election_timer: timeout(5000ms) STOPPED
vote_timer: timeout(5000ms) STOPPED
stepdown_timer: timeout(5000ms) SCHEDULING(in 3382ms)
snapshot_timer: timeout(600000ms) SCHEDULING(in 474855ms)
storage: [1, 153]
disk_index: 153
known_applied_index: 153
last_log_id: (index=153,term=2)
first_index_pinned: 1
state_machine: Idle
last_committed_index: 153
last_snapshot_index: 1
last_snapshot_term: 2
snapshot_status: IDLE
replicator_1116691496965@xxx: next_index=154   flying_append_entries_size=0 idle hc=262 ac=154 ic=0
replicator_3311419785217@xxx: next_index=154   flying_append_entries_size=0 idle hc=262 ac=154 ic=0

[Recovery From OTS Status]
state: FINISH
[Recovery From OTS Status]
state: FINISH
total rows: 1484409
table `jfs_block_test` 2 rows.
table `jfs_namespace_cache_ns` 1 rows.
table `jfs_namespace_test` 1484406 rows.

    7. Optional: Verify that the number of files on the new cluster is consistent with that on the original cluster.

      The cluster is now in read-only recovery mode. 

      # Compare the file count to verify that it is consistent with that of the original cluster.
[hadoop@emr-header-1 ~]$ hadoop fs -count jfs://test/
        1596      1482809                 25 jfs://test/
# Files can be read normally by using the cat or get command.
[hadoop@emr-header-1 ~]$ hadoop fs -cat jfs://test/testfile
this is a test file
# View the directory contents.
[hadoop@emr-header-1 ~]$ hadoop fs -ls jfs://test/
Found 3 items
drwxrwxr-x   - root   root            0 2020-03-25 14:54 jfs://test/emr-header-1.cluster-50087
-rw-r-----   1 hadoop hadoop          5 2020-03-25 14:50 jfs://test/haha-12096RANDOM.txt
-rw-r-----   1 hadoop hadoop         20 2020-03-25 15:07 jfs://test/testfile
# Files cannot be modified because the cluster is in read-only mode.
[hadoop@emr-header-1 ~]$ hadoop fs -rm jfs://test/testfile
java.io.IOException: ErrorCode : 25021 , ErrorMsg: Namespace is under recovery mode, and is read-only.

    8. Update the configuration to switch the cluster to normal mode and enable asynchronous data upload to Tablestore.

      On the bigboot tab of the SmartData service, set the following parameters.

      Parameter

      Description

      Example

      namespace.backend.raft.async.ots.enabled

      Specifies whether to enable asynchronous data upload to Tablestore. Valid values:

      • true

      • false

      true

      namespace.backend.raft.recovery.mode

      Specifies whether to enable metadata recovery from Tablestore. Valid values:

      • true

      • false

      false

    9. Restart the cluster.

      1. Click the Clusters tab.

      2. On the Clusters page, find the target cluster and choose More > Restart in the Actions column.