E-MapReduce:JindoFS cache mode

Configuration

OSS scheme (recommended)

The OSS scheme uses the standard oss://<bucket_name>/<path_of_your_file> path format. It works out of the box after you create an E-MapReduce cluster, with no additional configuration required. Existing jobs that read from or write to OSS run without modification.

JFS scheme

1. Go to the SmartData service.
   1. Log on to the EMR console.
      
   2. In the top navigation bar, select a region and a resource group as needed.
      
   3. Click the Clusters tab.
      
   4. On the Clusters page, find your cluster and click Details in the Actions column.
      
   5. In the navigation pane on the left, choose Services > SmartData.
2. Go to the bigboot service configuration page.
   1. Click the Configure tab.
   2. Click bigboot .
3. Configure the following parameters.
   JindoFS supports multiple namespaces. This topic uses test as an example namespace.
   1. Set jfs.namespaces to test.
      test is an example namespace name. To configure multiple namespaces, separate their names with commas (,).
   2. Click Custom Configuration. In the Add Configuration Item dialog box, add the following parameters.

      Parameter	Description	Example
      jfs.namespaces.test.oss.uri	The storage backend for the test namespace.	oss://<oss_bucket>/<oss_dir>/ Note You must set this parameter to a specific directory in an OSS bucket or to the root directory.
      jfs.namespaces.test.mode	Specifies that the test namespace uses cache mode.	cache

4. In the upper-right corner, click Save.
5. In the upper-right corner, choose Actions > > Restart Jindo Namespace Service.
   After the restart, you can access files by using the jfs://test/<path_to_your_file> path format. Files in this namespace are organized based on the root directory specified in jfs.namespaces.test.oss.uri. For example, jfs://test/hello.txt corresponds to the actual OSS object at oss://<oss_bucket>/<oss_dir>/hello.txt.

Enable cache

JindoFS can use local disks to cache frequently accessed data blocks. Caching is disabled by default, and all read operations access OSS directly.

1. On the Configure tab of the Services > SmartData service, click the client tab.
2. Set the jfs.cache.data-cache.enable parameter to 1 to enable caching.
   This is a client-side configuration and does not require you to restart the SmartData service.

After caching is enabled, the Jindo service automatically manages the local cache and evicts data based on the configured watermark. Configure the storage ratio based on your needs. For more information, see Control disk space by using watermarks.

Disk space watermark control

JindoFS uses OSS as its storage backend, which provides vast storage capacity. However, the local disk space on your cluster is finite. To manage this space, JindoFS automatically evicts cold data from the local cache. You can control this eviction behavior with the storage.watermark.high.ratio and storage.watermark.low.ratio parameters. These parameters accept decimal values between 0 and 1 that represent the ratio of disk space used.

1. Configure the disk watermarks.
   In the Service Configuration section, click the storage tab and modify the following parameters.

   Parameter	Description
   storage.watermark.high.ratio	The high watermark for disk usage. When the disk space used by the JindoFS data directory on a data disk reaches this ratio, JindoFS triggers a cleanup process. Default value: 0.4.
   storage.watermark.low.ratio	The low watermark for disk usage. After the cleanup process is triggered, JindoFS deletes cold data until the disk space used by the JindoFS data directory drops to this ratio. Default value: 0.2.

   Note You can set the high watermark to control the amount of disk space allocated to JindoFS. The low watermark must be less than the high watermark. Set reasonable values for your workload.
2. Save the configuration.
   1. In the upper-right corner, click Save.
   2. In the Confirm dialog box, enter a reason for the change, and turn on Auto-update Configuration.
   3. Click OK.
3. Restart the Jindo Storage Service to apply the changes.
   1. In the upper-right corner, choose Actions > Restart Jindo Storage Service.
   2. In the Cluster Activities dialog box, set the required parameters and click OK.
   3. In the Confirm dialog box, click OK.

Access OSS buckets

Advanced settings

These client-side configurations take effect without restarting the SmartData service.

• In the Service Configuration section, on the client tab, configure the following parameters.

  Parameter	Description
  client.oss.upload.threads	The number of OSS upload threads for each file write stream. Default: 4.
  client.oss.upload.max.parallelism	The maximum number of concurrent OSS uploads per process. Limits bandwidth and memory consumption from upload threads. Default: 16.

• In the Service Configuration section, on the smartdata-site tab, configure the following parameters.

  Parameter	Description
  fs.jfs.cache.copy.simple.max.byte	The maximum file size for standard copy during a rename. Files below this threshold use the standard copy interface; larger files use the multipart copy interface for better efficiency. Note If you have enabled the OSS fast copy feature, set this parameter to -1 to use the standard copy interface for all file sizes, leveraging fast copy for optimal rename performance.
  fs.jfs.cache.write.buffer.size	The buffer size for file write streams. Must be a power of two, up to 8 MB. Reduce this value if concurrent write streams cause high memory usage. Default: 1048576.
  fs.oss.committer.magic.enabled	Enables the Jindo Job Committer, which improves performance by avoiding rename operations during the job commit phase. Default: true. Note Object storage systems like OSS perform poorly during rename operations. The Jindo Job Committer addresses this in cache mode.