E-MapReduce:FAQ about AccessKeys for accessing OSS/OSS-HDFS

How do I access OSS or OSS-HDFS in an EMR cluster in password-free mode?

When you create an EMR cluster, an Elastic Compute Service (ECS) application role is automatically associated with the cluster. By default, this role is AliyunECSInstanceForEMRRole. JindoSDK uses this role to obtain a security token to access Object Storage Service (OSS) or OSS-HDFS.

To enable password-free access, confirm that EcsStsCredentialsProvider is configured in core-site.xml:

1. Go to the Configure tab of the Hadoop-Common service page and click the core-site.xml tab.

2. Check whether the value of fs.oss.credentials.provider contains com.aliyun.jindodata.oss.auth.EcsStsCredentialsProvider.

3. If the value is missing, add com.aliyun.jindodata.oss.auth.EcsStsCredentialsProvider to the parameter value and click Save.

How do I check whether password-free access to OSS or OSS-HDFS is available in an EMR cluster?

Option 1: Verify that the RAM role returns valid credentials

Run the following command on a cluster node to query the RAM role credentials:

curl http://100.100.100.200/latest/meta-data/Ram/Security-credentials/AliyunECSInstanceForEMRRole

If the response includes "Code": "Success" and a valid SecurityToken, password-free access is available:

{
  "AccessKeyId": "STS.NUreXXXXXX",
  "AccessKeySecret": "BsmbnDoXXXXXXXX",
  "Expiration": "2022-11-22T11:27:39Z",
  "SecurityToken": "CAISlwJ1q6FXXXXXXX",
  "LastUpdated": "2022-11-22T05:27:39Z",
  "Code": "Success"
}

Option 2: Test access by using Hadoop shell commands

1. Go to the Configure tab of the Hadoop-Common service page and click the core-site.xml tab.

2. Confirm that fs.oss.credentials.provider contains com.aliyun.jindodata.oss.auth.EcsStsCredentialsProvider. If the value is missing, add it and click Save.

3. Run Hadoop Distributed File System (HDFS) shell commands to access OSS or OSS-HDFS. If the commands succeed, password-free access is working.

What do I do if I am unable to access OSS or OSS-HDFS by using HiveServer2 or Presto every 5 to 6 hours?

In some versions of JindoData, some known issues about password-free access exist. See Known issues in JindoData 4.X for the full list of affected versions.

Method 1: Use a fixed AccessKey pair

Configure SimpleCredentialsProvider to authenticate with a fixed AccessKey pair. See Configure a credential provider for OSS or OSS-HDFS.

Method 2: Upgrade JindoSDK

Upgrade JindoSDK to the latest version, which includes fixes for known password-free access issues. See Upgrade JindoSDK in EMR clusters.

What do I do if an error indicating that the OSS or OSS-HDFS path contains an AccessKey pair is reported?

Error message:

The Filesystem URI contains login details. This authentication mechanism is no longer supported.

JindoSDK V4.0.0 and later versions reject AccessKey pairs embedded in the filesystem URI to prevent credential leaks.

Option 1: Remove the AccessKey pair from the URI (recommended)

Update your application or configuration to use a credential provider instead of embedding the AccessKey pair in the access path.

Option 2: Re-enable URI-based authentication

If your use case requires embedding credentials in the URI, add the following configuration item on the core-site.xml tab of the Hadoop-Common service page:

Click Add Configuration Item, enter the values above, and then click Save.