Configure an OSS/OSS-HDFS credential provider - E-MapReduce

This topic explains how to configure an OSS/OSS-HDFS credential provider.

Basic configuration

You can pre-configure the AccessKey ID, AccessKey secret, and endpoint for OSS or OSS-HDFS in the Hadoop core-site.xml file. To configure these parameters for an EMR cluster, follow these steps:

On the Hadoop-Common service configuration page, click the core-site.xml tab.
Click Add Configuration Item, and add the configuration items fs.oss.accessKeyId and fs.oss.accessKeySecret.
Click OK, enter the execution reason, and click OK.
Click Save below.

Store AccessKeys with Hadoop credential providers

In the basic configuration method, fs.oss.accessKeyId and fs.oss.accessKeySecret are stored in plaintext in core-site.xml. You can store them as an encrypted pair in a Hadoop Credential Providers file.

Store the AccessKey ID, AccessKey secret, and security token in a credential file by using the following Hadoop command:

hadoop credential <subcommand> [options]

In addition to file permissions, you can also specify a password to encrypt the credentials. If you omit the password, the system uses a default string for encryption.

hadoop credential create fs.oss.accessKeyId -value AAA -provider jceks://file/root/oss.jceks
hadoop credential create fs.oss.accessKeySecret -value BBB -provider jceks://file/root/oss.jceks
hadoop credential create fs.oss.securityToken -value CCC -provider jceks://file/root/oss.jceks

After generating the credential file, configure the following property to specify the provider type and location:

<configuration>
    <property>
        <name>fs.oss.security.credential.provider.path</name>
        <value>jceks://file/root/oss.jceks</value>
        <description>Specifies the credential file that stores your access keys. For example, jceks://file/${user.home}/oss.jceks points to the oss.jceks file in your home directory.</description>
    </property>
</configuration>

Use JindoSDK credential providers

By default, JindoSDK is configured with SimpleCredentialsProvider, EnvironmentVariableCredentialsProvider, and CommonCredentialsProvider. The system queries each provider sequentially until it retrieves valid credentials.

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.SimpleCredentialsProvider,com.aliyun.jindodata.oss.auth.EnvironmentVariableCredentialsProvider,com.aliyun.jindodata.oss.auth.CommonCredentialsProvider</value>
        <description>Specifies the com.aliyun.jindodata.oss.auth.JindoCredentialsProvider. To use multiple providers, separate the class names with a comma (,). The system attempts to retrieve credentials from each provider in order. For more information about the providers, see Credential provider types.</description>
    </property>
</configuration>

Select a credential provider from the following options:

Credential provider type	Description
TemporaryCredentialsProvider	Use this provider to access OSS or OSS-HDFS with a time-bound AccessKey ID, AccessKey secret, and security token.
SimpleCredentialsProvider	Use this provider to access OSS or OSS-HDFS with a long-term AccessKey ID and AccessKey secret.
EnvironmentVariableCredentialsProvider	Retrieves credentials from environment variables.
CommonCredentialsProvider	A provider that retrieves credentials from common Jindo configuration parameters.
CustomCredentialsProvider	Integrates with a custom password-free service.

TemporaryCredentialsProvider

Configure the provider type

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.TemporaryCredentialsProvider</value>
    </property>
</configuration>

Configure OSS or OSS-HDFS credentials

<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>YOUR_ACCESS_KEY_ID</value>
    </property>
    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>YOUR_ACCESS_KEY_SECRET</value>
    </property>
    <property>
        <name>fs.oss.securityToken</name>
        <value>YOUR_SECURITY_TOKEN</value>
    </property>
</configuration>

SimpleCredentialsProvider

Configure the provider type

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.SimpleCredentialsProvider</value>
    </property>
</configuration>

Configure OSS or OSS-HDFS credentials

<configuration>
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>YOUR_ACCESS_KEY_ID</value>
    </property>
    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>YOUR_ACCESS_KEY_SECRET</value>
    </property>
</configuration>

EnvironmentVariableCredentialsProvider

Configure the provider type

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.EnvironmentVariableCredentialsProvider</value>
    </property>
</configuration>

Configure OSS or OSS-HDFS credentials

Set the following environment variables:

Parameter	Description
OSS_ACCESS_KEY_ID	The AccessKey ID for OSS or OSS-HDFS.
OSS_ACCESS_KEY_SECRET	The AccessKey secret for OSS or OSS-HDFS.
OSS_SECURITY_TOKEN	The security token for OSS or OSS-HDFS. Note Required only when using a time-bound token.

CommonCredentialsProvider

Configure the provider type

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.CommonCredentialsProvider</value>
    </property>
</configuration>

Configure OSS or OSS-HDFS credentials

<configuration>
    <property>
        <name>jindo.common.accessKeyId</name>
        <value>YOUR_ACCESS_KEY_ID</value>
    </property>
    <property>
        <name>jindo.common.accessKeySecret</name>
        <value>YOUR_ACCESS_KEY_SECRET</value>
    </property>
    <property>
        <name>jindo.common.securityToken</name>
        <value>YOUR_SECURITY_TOKEN. (Required for time-bound tokens)</value>
    </property>
</configuration>

CustomCredentialsProvider

Configure the provider type

<configuration>
    <property>
        <name>fs.oss.credentials.provider</name>
        <value>com.aliyun.jindodata.oss.auth.CustomCredentialsProvider</value>
    </property>
</configuration>

Configure the password-free service URL
```
<configuration>
    <property>
        <name>aliyun.oss.provider.url</name>
        <value>The URL of the password-free service.</value>
    </property>
</configuration>
```
aliyun.oss.provider.url supports the http(s) and Secrets protocols:
- HTTP(S) protocol
  The URL format for the HTTP(S) password-free service is http://localhost:1234/sts. The HTTP password-free protocol requires the response to be in JSON format.
```
{
"AccessKeyId" : "XXXXXXXXX",
"AccessKeySecret" : "XXXXXXXXX",
"Expiration" : "2020-11-01T05:20:01Z",
"SecurityToken" : "XXXXXXXXX",
"LastUpdated" : "2020-10-31T23:20:01Z",
"Code" : "Success"
}
```
- Secrets protocol
  The address format for the Secrets password-free service is secrets:///local_path_prefix, which is commonly used in Kubernetes scenarios. Here, local_path_prefix is the path prefix:
  - If local_path_prefix is secrets:///secret/JindoOss, files such as /secret/JindoOssAccessKeyId, /secret/JindoOssAccessKeySecret, and /secret/JindoOssSecurityToken are searched for on the node.
  - If local_path_prefix is secrets:///secret/JindoOss/, files such as /secret/JindoOss/AccessKeyId, /secret/JindoOss/AccessKeySecret, and /secret/JindoOss/SecurityToken will be searched for on the node.