All Products
Search

This Product

    E-MapReduce:Use JindoFS in cache mode

    Document Center

    E-MapReduce:Use JindoFS in cache mode

    Last Updated:Mar 26, 2026

    In cache mode, JindoFS stores files as objects in Object Storage Service (OSS) and caches frequently accessed files on local EMR cluster disks. This improves read and write performance and reduces bandwidth pressure, without requiring data migration or format conversion — your existing OSS clients and jobs continue to work as-is.

    Choose an access method

    Cache mode supports two access methods:

    MethodSchemeWhen to use
    OSS Scheme (recommended)oss://<bucket_name>/<path>Your jobs already use OSS paths. No extra configuration needed after cluster creation.
    JFS Schemejfs://test/<path>You need JindoFS namespace management. Requires additional namespace setup.

    Use OSS Scheme unless you specifically need independent namespace management in JindoFS.

    Prerequisites

    Before you begin, ensure that you have:

    • An EMR cluster

    • Access to the SmartData service in the EMR console

    • (For cross-account or cross-region OSS access) An AccessKey ID and AccessKey secret with access to the target OSS bucket

    Configure JFS Scheme

    Skip this section if you are using OSS Scheme.

    1. Go to the SmartData service.

      1. Log on to the Alibaba Cloud EMR console.

      2. In the top navigation bar, select the region where your cluster resides. Select the resource group as required. By default, all resources of the account appear.

      3. Click the Cluster Management tab.

      4. On the Cluster Management page, find the target cluster and click Details in the Actions column.

      5. In the left-side navigation pane, click Cluster Service > SmartData.

    2. On the Configure tab, click the namespace tab in the Service Configuration section.

      namespace

    3. Set jfs.namespaces to test. To configure multiple namespaces, separate them with commas (,). This topic uses a namespace named test as an example.

    4. Click Custom Configuration. In the Add Configuration Item dialog box, configure the following parameters and click OK.

      ParameterDescriptionExample
      jfs.namespaces.test.oss.uriThe storage backend of the test namespace. Set this to a specific OSS bucket directory or the root directory.oss://<oss_bucket>/<oss_dir>/
      jfs.namespaces.test.modeThe storage mode.cache

    5. Click OK.

    6. Save the configuration.

      1. In the upper-right corner of the Service Configuration section, click Save.

      2. In the Confirm Changes dialog box, enter a description and turn on Auto-update Configuration.

      3. Click OK.

    7. From the Actions drop-down list in the upper-right corner, select Restart Jindo Namespace Service.

    After Namespace Service restarts, access files using jfs://test/<path_of_file>.

    Enable local cache

    By default, local cache is disabled and EMR reads data directly from OSS. When enabled, hot data blocks are cached on local disks.

    1. On the Configure tab of the SmartData service page, click the client tab.

    2. Set jfs.cache.data-cache.enable to true.

    3. Save the configuration.

      1. In the upper-right corner of the Service Configuration section, click Save.

      2. In the Confirm Changes dialog box, enter a description and turn on Auto-update Configuration.

      3. Click OK.

    No service restart is required. JindoFS automatically manages cached data after you enable this setting.

    Control disk space usage

    JindoFS automatically evicts cold data from local disks when usage is high. The high and low watermarks control when eviction starts and stops: eviction begins when disk usage exceeds the high watermark and stops when usage falls to the low watermark.

    On the Configure tab of the SmartData service page, click the storage tab.

    storage

    Configure the following parameters:

    ParameterDescriptionDefault
    storage.watermark.high.ratioStart evicting when JindoFS disk usage exceeds this ratio.0.4
    storage.watermark.low.ratioStop evicting when JindoFS disk usage falls to this ratio.0.2

    Set both values to decimals between 0 and 1. The high watermark must be greater than the low watermark.

    After configuring, save and restart Jindo Storage Service:

    1. In the upper-right corner of the Service Configuration section, click Save.

    2. In the Confirm Changes dialog box, enter a description and turn on Auto-update Configuration.

    3. Click OK.

    4. From the Actions drop-down list, select Restart Jindo Storage Service.

    5. In the Cluster Activities dialog box, configure the related parameters and click OK.

    6. In the confirmation message, click OK.

    Access an OSS bucket in a different account or region

    If the OSS bucket and your EMR cluster are in the same Alibaba Cloud account and region, no AccessKey configuration is required. For cross-account or cross-region access, configure an AccessKey pair and endpoint.

    OSS Scheme

    1. On the Configure tab of the SmartData service page, click the smartdata-site tab.

    2. Click Custom Configuration.

    3. In the Add Configuration Item dialog box, add the following parameters and click OK.

      ParameterDescription
      fs.jfs.cache.oss.accessKeyIdAccessKey ID for the OSS bucket.
      fs.jfs.cache.oss.accessKeySecretAccessKey secret for the OSS bucket.
      fs.jfs.cache.oss.endpointEndpoint of the OSS bucket.
    Note These parameters are also compatible with EMR versions earlier than V3.30.0.

    JFS Scheme

    1. On the Configure tab of the SmartData service page, click the namespace tab.

    2. Set jfs.namespaces to test.

    3. Click Custom Configuration in the upper-right corner.

    4. In the Add Configuration Item dialog box, add the following parameters and click OK.

      ParameterDescription
      jfs.namespaces.test.oss.uriStorage backend of the test namespace. Include the endpoint in the URI: oss://<oss_bucket.endpoint>/<oss_dir>.
      jfs.namespaces.test.oss.access.keyAccessKey ID for the OSS bucket.
      jfs.namespaces.test.oss.access.secretAccessKey secret for the OSS bucket.

    Configure advanced parameters

    These optional parameters tune upload and write performance. No restart is required after changes.

    Upload concurrency (client tab)

    On the client tab, configure the following parameters:

    ParameterDescriptionDefault
    client.oss.upload.threadsNumber of OSS upload threads per write stream.4
    client.oss.upload.max.parallelismMaximum concurrent OSS upload threads per process. Reduce this if upload threads consume too much bandwidth or memory.16

    Write buffer and job committer (smartdata-site tab)

    On the smartdata-site tab, configure the following parameters:

    ParameterDescriptionDefault
    fs.jfs.cache.write.buffer.sizeBuffer size for write streams, in bytes. Must be a power of 2. Maximum: 8,388,608 (8 MB). Reduce this if write streams use too much memory.1048576
    fs.oss.committer.magic.enabledEnable Jindo Job Committer, which skips rename operations to improve job commit performance. In cache mode, rename operations on OSS are slower than standard — keep this enabled unless you have a specific reason to disable it.true