E-MapReduce:FAQ: AccessKey pair issues when accessing OSS or OSS-HDFS

• How to enable password-free access to OSS or OSS-HDFS

• How to check if password-free access is available in an EMR cluster

• HiveServer2 or Presto fails to access OSS or OSS-HDFS every 5 to 6 hours

• An error occurs when an access path to OSS or OSS-HDFS contains an AccessKey pair

Enable password-free access to OSS or OSS-HDFS

On the Hadoop-Common service configuration page, go to the core-site.xml tab. Verify that the value of the fs.oss.credentials.provider parameter is com.aliyun.jindodata.oss.auth.EcsStsCredentialsProvider. If not, set the parameter to this value and click Save. This enables password-free access to OSS or OSS-HDFS.

Check for password-free access

• Method 1: Run the following command to retrieve the RAM role credentials.

  curl http://100.100.100.200/latest/meta-data/Ram/Security-credentials/AliyunECSInstanceForEMRRole

  If the command returns a result in the following format, password-free access is available.

   {
    "AccessKeyId" : "STS.NUreXXXXXX",
    "AccessKeySecret" : "BsmbnDoXXXXXXXX",
    "Expiration" : "2022-11-22T11:27:39Z",
    "SecurityToken" : "CAISlwJ1q6FXXXXXXX",
    "LastUpdated" : "2022-11-22T05:27:39Z",
    "Code" : "Success"
    }

• Method 2: Configure and test password-free access.

  1. On the Hadoop-Common service configuration page, go to the core-site.xml tab. Verify that the value of the fs.oss.credentials.provider parameter is com.aliyun.jindodata.oss.auth.EcsStsCredentialsProvider. If not, set the parameter to this value and click Save.

  2. Use the HDFS shell to access OSS or OSS-HDFS. Successful access confirms that password-free access is available.

Periodic access failures from HiveServer2 or Presto

Certain JindoSDK versions have a known bug that causes password-free access to fail periodically. For details, see Known issues in JindoData.

• Method 1: Use a static AccessKey pair.

  Configure your system to use SimpleCredentialsProvider. For more information, see Configure an OSS/OSS-HDFS Credential Provider.

• Method 2: Upgrade JindoSDK to the latest version. For more information, see Upgrade and roll back JindoSDK on an EMR cluster (x86).

Error: AccessKey pair in access path

• Error message:

  The Filesystem URI contains login details. This authentication mechanism is no longer supported.

• Cause:

  To prevent credential leakage, JindoSDK 4.0.0 and later do not allow AccessKey pairs (AccessKey ID and AccessKey Secret) in the access path.

• Solutions:

  • Method 1: Remove the AccessKey pair from the access path.

  • Method 2: If you must include the AccessKey pair in the path, go to the Hadoop-Common service configuration page, click the core-site.xml tab, and then click Add Configuration Item. Add the following configuration:

    Parameter	Value
    fs.oss.uri-with-secrets.enable	true

    After you add the configuration, click Save.