JindoFS cache mode stores your data as objects in Object Storage Service (OSS) while caching frequently accessed files on local EMR cluster disks. This lets you read and write OSS data at local disk speeds without migrating data or converting file formats.
How cache mode works
Cache mode follows a three-phase lifecycle:
Load: JindoFS reads frequently accessed files from OSS and caches them on a local disk.
Manage: JindoFS automatically tracks which cached files are hot. Local disk space is controlled by configurable high and low watermarks.
Evict: When disk usage exceeds the high watermark, JindoFS evicts cold data until usage drops to the low watermark.
By default, local cache is disabled and all reads go directly to OSS. Enable it when read-heavy workloads need lower latency.
Choose an access scheme
Cache mode supports two ways to access OSS:
| OSS Scheme | JFS Scheme | |
|---|---|---|
| URI format | oss://<bucket>/<path> | jfs://<namespace>/<path> |
| Setup required | None — works immediately after cluster creation | Requires namespace configuration in SmartData |
| OSS client compatibility | Full — existing jobs run without changes | Requires URI updates |
| When to use | Default choice for most workloads | When you need namespace-level access control or multiple storage backends |
Use OSS Scheme unless you have a specific reason to use JFS Scheme.
Access OSS with OSS Scheme
No additional configuration is required. After creating an EMR cluster, access OSS directly:
oss://<bucket_name>/<path_of_your_file>Existing jobs that already use this URI format continue to work without modification.
Configure JFS Scheme
Prerequisites
Before you begin, ensure that you have:
An EMR cluster with SmartData service enabled
Access to the Alibaba Cloud E-MapReduce console
Configure a namespace
Log on to the Alibaba Cloud E-MapReduce console.
In the top navigation bar, select the region where your cluster resides. Select the resource group as required.
Click the Cluster Management tab. Find your cluster and click Details in the Actions column.
In the left-side navigation pane, click Cluster Service > SmartData.
Click the Configure tab. In the Service Configuration section, click the bigboot tab.
Set
jfs.namespacestotest.JindoFS supports multiple namespaces. Separate multiple namespace names with commas. This topic uses a namespace named
test.Click Custom Configuration. In the Add Configuration Item dialog box, configure the following parameters and click OK.
Parameter Description Example jfs.namespaces.test.oss.uriThe OSS storage backend for the testnamespace. Set this to a specific directory or the root directory of an OSS bucket.oss://<oss_bucket>/<oss_dir>/jfs.namespaces.test.modeThe storage mode. Set to cachefor cache mode.cacheIn the upper-right corner of the Service Configuration section, click Save.
From the Actions drop-down list in the upper-right corner, select Restart Jindo Namespace Service.
After the service restarts, access files using the JFS Scheme:
jfs://test/<path_to_your_file>JindoFS maps JFS paths to OSS paths based on jfs.namespaces.test.oss.uri. For example, jfs://test/hello.txt maps to oss://<oss_bucket>/<oss_dir>/hello.txt.
Enable local cache
By default, EMR reads data directly from OSS. Enable local cache to store hot data blocks on local disks for faster repeated reads.
In the left-side navigation pane, click Cluster Service > SmartData. On the SMARTDATA page, click the Configure tab. In the Service Configuration section, click the client tab.
Set
jfs.cache.data-cache.enableto1.
The change takes effect immediately on the client — no restart of the SmartData service is needed.
After you enable local cache, JindoFS automatically manages cached data using the high and low watermarks described in Control disk space usage.
Control disk space usage
JindoFS automatically evicts cold data when local disk usage is high. Two parameters control the eviction thresholds:
| Parameter | Description | Default |
|---|---|---|
storage.watermark.high.ratio | Upper limit of disk usage for JindoFS data. When usage exceeds this ratio, JindoFS starts evicting data. | 0.4 |
storage.watermark.low.ratio | Lower limit. JindoFS continues evicting data until disk usage drops to this ratio. | 0.2 |
Both parameters accept decimal values between 0 and 1. The high ratio must be greater than the low ratio.
To update the watermark settings:
In the Service Configuration section for SmartData, click the storage tab.
Update
storage.watermark.high.ratioandstorage.watermark.low.ratioas needed.In the upper-right corner of the Service Configuration section, click Save. In the Confirm Changes dialog box, specify a description, turn on Auto-update Configuration, and click OK.
From the Actions drop-down list in the upper-right corner, select Restart Jindo Storage Service. In the Cluster Activities dialog box, specify the required parameters and click OK. In the confirmation message, click OK.
Access a cross-account or cross-region OSS bucket
If your EMR cluster and OSS bucket are in the same Alibaba Cloud account and the same region, no AccessKey configuration is needed.
For all other cases (different account or different region), configure an AccessKey pair and endpoint explicitly.
OSS Scheme
In the left-side navigation pane, click Cluster Service > SmartData. On the SMARTDATA page, click the Configure tab. In the Service Configuration section, click the smartdata-site tab.
Click Custom Configuration. In the Add Configuration Item dialog box, configure the following parameters and click OK.
Parameter Description fs.jfs.cache.oss-accessKeyIdThe AccessKey ID of the OSS bucket. fs.jfs.cache.oss-accessKeySecretThe AccessKey secret of the OSS bucket. fs.jfs.cache.oss-endpointThe endpoint of the OSS bucket.
JFS Scheme
In the left-side navigation pane, click Cluster Service > SmartData. On the SMARTDATA page, click the Configure tab. In the Service Configuration section, click the bigboot tab.
Set
jfs.namespacestotest.Click Custom Configuration. In the Add Configuration Item dialog box, configure the following parameters and click OK.
Parameter Description jfs.namespaces.test.oss.uriThe OSS storage backend URI. Include the endpoint in the URI. Example: oss://<oss_bucket.endpoint>/<oss_dir>.jfs.namespaces.test.oss.access.keyThe AccessKey ID of the OSS bucket. jfs.namespaces.test.oss.access.secretThe AccessKey secret of the OSS bucket.
Advanced configurations
The following parameters tune cache performance. Changes take effect immediately on the client — no service restart is needed.
client tab (in the Service Configuration section):
| Parameter | Description | Default |
|---|---|---|
client.oss.upload.threads | Number of OSS upload threads per data write stream. | 4 |
client.oss.upload.max.parallelism | Maximum concurrent OSS upload threads per process. This cap prevents upload threads from consuming excessive bandwidth and memory. | 16 |
smartdata-site tab (in the Service Configuration section):
| Parameter | Description | Default |
|---|---|---|
fs.jfs.cache.copy.simple.max.byte | File size threshold for rename operations. Files smaller than this value use the common copy interface; larger files use Multipart Copy for better performance. If OSS fast copy is enabled, set to -1 to use the common copy interface for all files and get optimal rename performance. | — |
fs.jfs.cache.write.buffer.size | Buffer size for data write streams, in bytes. Must be a power of 2. Maximum value: 8388608 (8 MB). Reduce this value if write streams consume too much memory. | 1048576 |
fs.oss.committer.magic.enabled | Enables Jindo Job Committer, which commits jobs without rename operations and improves commit performance. In cache mode, OSS rename performance is lower than standard, so Jindo Job Committer is the recommended committer. | true |