JindoFS OSS configuration

Prerequisites

A high-security E-MapReduce (EMR) cluster is created and the Ranger service is selected when you create the cluster. For more information about how to create a cluster, see Create a cluster.

Note You must turn on Kerberos Mode in the Advanced Settings section of the Software Settings step when you create the cluster. Otherwise, the cluster is not a high-security cluster.

Limits

Integrate OSS with Ranger

1. Go to the Cluster Overview page.
   1. Log on to the Alibaba Cloud EMR console.
   2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
   3. Click the Cluster Management tab.
   4. On the Cluster Management page, find your cluster and click Details in the Actions column.
2. Enable OSS.
   1. In the left-side navigation pane, choose Cluster Service > RANGER.
   2. In the upper-right corner of the RANGER service page, choose Actions > EnabledOSS.
   3. In the Cluster Activities dialog box, configure Description and click OK.
   4. In the Confirm message, click OK.
3. Deploy client configurations.
   • EMR V5.6.0 or a later minor version, or EMR V3.40.0 or a later minor version
     1. In the left-side navigation pane, choose Cluster Service > HDFS.
     2. On the HDFS service page, click the Configure tab. In the upper-right corner of the Service Configuration section, click Deploy Client Configuration.
     3. In the Cluster Activities dialog box, configure Description and click OK.
     4. In the Confirm message, click OK.

        You can click History in the upper-right corner to view the execution status and progress.

   • EMR V5.4.2 or EMR V3.38.2
     1. In the left-side navigation pane, choose Cluster Service > SmartData.
     2. On the SmartData service page, click the Configure tab. In the upper-right corner of the Service Configuration section, click Deploy Client Configuration.
     3. In the Cluster Activities dialog box, configure Description and click OK.
     4. In the Confirm message, click OK.

        You can click History in the upper-right corner to view the execution status and progress.

4. Restart Jindofsx Namespace Service or Jindo Namespace Service.
   • EMR V5.6.0 or a later minor version, or EMR V3.40.0 or a later minor version
     1. In the upper-right corner of the JindoData service page, choose Actions > Restart Jindofsx Namespace Service.
     2. In the Cluster Activities dialog box, configure Description and click OK.
     3. In the Confirm message, click OK.
   • EMR V5.4.2 or EMR V3.38.2
     1. In the upper-right corner of the SmartData service page, choose Actions > Restart Jindo Namespace Service.
     2. In the Cluster Activities dialog box, configure Description and click OK.
     3. In the Confirm message, click OK.
5. Restart HiveServer2.
   1. In the left-side navigation pane, choose Cluster Service > Hive.
   2. In the upper-right corner of the Hive service page, choose Actions > Restart HiveServer2.
   3. In the Cluster Activities dialog box, configure Description and click OK.
   4. In the Confirm message, click OK.
6. Create a principal.
   1. Log on to the emr-header-1 node of your cluster in SSH mode. For more information, see Log on to a cluster.
   2. Run the following command to enable the Kerberos administration tool:

      sh /usr/lib/has-current/bin/admin-local.sh /etc/ecm/has-conf -k /etc/ecm/has-conf/admin.keytab

   3. Run the following command to create a principal named test.
      In this example, the password is 123456.

      addprinc -pw 123456 test

      Note You must record the username and password, which will be used to create a Ticket Granting Ticket (TGT). If you do not want to record the username and password, you can perform the next step to generate a keytab file and import the username and password of the principal into the keytab file.
   4. Optional:Run the following command to generate a keytab file:

      ktadd -k /root/test.keytab test

      To exit the Kerberos administration tool, run the quit command.

7. Create a TGT.
   You can create a TGT on one of the nodes on which you want to run a Hive client.
   1. Run the following command as the root user to create a user named test:

      useradd test

   2. Run the following command to switch to the test user:

      su test

   3. Create a TGT.
      • Method 1: Use a username and password to create a TGT.
        Enter kinit and press Enter. Then, enter the password 123456 of the test account.
      • Method 2: Use a keytab file to create a TGT.
        The test.keytab file generated in Step 6 is stored in the /root/ directory of the emr-header-1 node. You must run the cp /root/test.keytab /home/test/ command to copy the file to the /home/test/ directory of the current node. Then, run the following command to create a TGT:

        kinit -kt /home/test/test.keytab test

   4. View the information about the TGT.
      Run the klist command. The following information is returned:

      Ticket cache: FILE:/tmp/krb5cc_1012
      Default principal: test@EMR.23****.COM
      
      Valid starting       Expires              Service principal
      07/24/2021 13:20:44  07/25/2021 13:20:44  krbtgt/EMR.23****.COM@EMR.23****.COM
              renew until 07/25/2021 13:20:44

Permission configuration example

In this topic, an EMR V3.38.2 cluster is used. For clusters of other versions, the actual operations on the Ranger web UI prevail.

Perform the following steps to grant the test user all the access permissions on the oss://bucket-test-hangzhou/user/test directory.

1. Access the Ranger web UI. For more information, see Overview.
2. On the Ranger web UI, click emr-oss.
   
3. Grant all access permissions on the test directory.
   1. Click Add New Policy in the upper-right corner.
   2. On the Edit Policy page, configure the parameters that are described in the following table.
      

      Parameter	Description
      Policy Name	The name of the policy. You can specify a custom name.
      Path	The OSS path. You do not need to start the path with the oss:// prefix. In this example, set this parameter to bucket-test-hangzhou/user/test. Notice No forward slash (/) is required at the end of the path. Do not turn off recursive.
      Select User	The user to whom you want to attach this policy. In this example, set this parameter to test.
      Permissions	The permissions that you want to grant. In this example, select ALL. ALL indicates the Read, Write, and Execute permissions.

   3. Click Add.
4. Grant the Execute permission on the user directory, which is the parent directory of the test directory.
   1. Click Add New Policy in the upper-right corner.
   2. On the Edit Policy page, configure the parameters that are described in the following table.
      

      Parameter	Description
      Policy Name	The name of the policy. You can specify a custom name.
      Path	The OSS path. You do not need to start the path with the oss:// prefix. In this example, set this parameter to bucket-test-hangzhou/user. Notice No forward slash (/) is required at the end of the path. Do not turn off recursive.
      Select User	The user to whom you want to attach this policy. In this example, set this parameter to test.
      Permissions	The permissions that you want to grant. In this example, set this parameter to Execute.

   3. Click Add.
5. Access OSS.
   1. Log on to the emr-header-1 node of your cluster in SSH mode. For more information, see Log on to a cluster.
   2. Run the following command to switch to the test user:

      su test

   3. Run the following command to access the OSS directory:

      hadoop fs -ls oss://bucket-test-hangzhou/user/test

      If you are not granted the permissions to access the OSS directory, the following information is displayed:

      org.apache.hadoop.security.AccessControlException: Permission denied: user=test, access=READ_EXECUTE, resourcePath="bucket-test-hangzhou/"