JindoSDK lets you read from and write to Object Storage Service (OSS) or OSS-HDFS from E-MapReduce (EMR) clusters and non-EMR Hadoop environments. OSS and OSS-HDFS use the same access methods — the only difference is the endpoint format in the path.
Prerequisites
Before you begin, ensure that you have:
-
An activated OSS service. See Activate OSS.
-
An OSS bucket. See Create buckets.
-
OSS access permissions granted to your account:
-
New EMR console: permissions are granted by default. See Assign roles.
-
Old EMR console: permissions are granted by default. See Assign roles.
-
Non-EMR environment: See Grant access to OSS or OSS-HDFS.
-
-
(Recommended) OSS-HDFS activated with access permissions granted.
-
JindoSDK deployed:
-
EMR clusters: JindoSDK is deployed automatically.
Note To access OSS-HDFS from an EMR cluster, create a cluster running EMR V3.42.0 or a later minor version, or EMR V5.8.0 or a later minor version. -
Non-EMR environments: Deploy JindoSDK manually. See Deploy JindoSDK in an environment other than EMR.
Note To access OSS-HDFS from a non-EMR environment, deploy JindoSDK 4.X or later.
-
Path formats
OSS and OSS-HDFS use different endpoints in the path. The table below shows sample root paths and when to use each storage system.
| Storage system | Sample root path | When to use |
|---|---|---|
| OSS | oss://examplebucket.oss-cn-shanghai-internal.aliyuncs.com/ |
General-purpose object storage. Use an internal endpoint for access within the same region. Nodes without public IP addresses cannot use a public endpoint, so cross-region access is not supported in that case. |
| OSS-HDFS | oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/ |
Accessible only via private IP address — cross-region access is not supported. |
oss:// for both storage systems. The endpoint portion of the path determines which system is accessed.Access methods
JindoSDK supports four ways to access OSS or OSS-HDFS. Choose the method that fits your workflow.
| Method | Example | Description |
|---|---|---|
| Hadoop Shell command | hadoop fs -ls oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/ |
JindoOssFileSystem in JindoSDK implements the Hadoop FileSystem interface. The endpoint in the path determines whether OSS or OSS-HDFS is accessed. See Use Hadoop Shell commands to access OSS or OSS-HDFS. |
| Jindo CLI command | jindo fs -ls oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/ |
Similar to Hadoop Shell, with additional operations such as archiving, caching, and error analysis. See Use Jindo CLI commands to access OSS or OSS-HDFS. |
| POSIX command | mkdir -p /mnt/ossjindo-fuse /mnt/oss -ouri=oss://examplebucket.cn-shanghai.oss-dls.aliyuncs.com/ls /mnt/oss |
JindoFuse mounts OSS or OSS-HDFS as a local path using the FUSE API. Access the storage the same way as a local file system. See Use POSIX commands to access OSS or OSS-HDFS. |
| OSS console |  |
Log on to the OSS console. In the left-side navigation pane, click Buckets, find your bucket, and click its name. Click the OSS Object or HDFS tab to access OSS or OSS-HDFS. |