Best practices for unified access control with Data Lake Formation (DLF) and E-MapReduce (EMR) - Data Lake Formation

Background

DLF provides a unified metadata view that resolves metadata consistency and data sharing issues. Compute engines that integrate with DLF also gain enterprise-grade access control capabilities.

Key concepts

EMR: E-MapReduce is an Alibaba Cloud service for big data processing. For more information, see What is E-MapReduce.
DLF Catalog: The top-level entity in the DLF metadata system. It can contain metadata for objects such as databases and tables. For more information, see data catalog.
DLF data permissions: The data permission system that DLF provides for data lakes. It supports fine-grained access control at four levels: database, table, column, and function. For more information, see Data permissions overview.

Business scenario

A company that uses an EMR cluster with multiple query engines (including Hive, Spark, Presto, and Impala) wants to implement a unified access control system to manage data access for different roles:

super administrator
- Has all permissions on the data lake and can grant them to other users.
business A data administrator
- Has full access to all data related to Business A in the db_a database. This user can also grant permissions on this database to other users.
business A data developer
- Has full access to all data in the db_a database.
business A data analyst
- Has access to specific columns in specific tables within the db_a database, such as the col1 and col2 columns in the table1 table.

Procedure

Create an EMR cluster that uses DLF for metadata management.
1. Log on to the E-MapReduce console.
2. Create an EMR cluster and specify the following options:
  - Business Scenario: Select Data Lake.
  - Optional Services: Select at least the Hive and DLF-Auth components. Select other components based on your business requirements.
  - Metadata: Select DLF Unified Metadata.
  - DLF Catalog: Select the default DLF Catalog or create a new one. This tutorial uses catalog_test as an example.
3. Configure other settings as required and create the EMR cluster.
  Note
  - If your existing E-MapReduce cluster does not have the DLF-Auth component, you can add it as a service to use DLF data permissions.
  - If you have an existing E-MapReduce cluster where the Hive metadata is not managed by DLF, you must first migrate the metadata before you can use DLF data permissions.You can also contact us through our DingTalk group: 33719678.

Initialize the required databases and tables.

Log on to the EMR cluster. For more information, see Log on to a cluster.

Connect to Hive by using Beeline.

beeline -u jdbc:hive2://<primary_node_name>:10000

Run the following statements to initialize and create test data.

-- Create databases and tables
create database db_a;
create table db_a.table1(
col1 string,
col2 string,
col3 string
);
create table db_a.table2(
col1 string,
col2 string,
col3 string
);

create database db_b;
create table db_b.table1(
col1 string,
col2 string,
col3 string
);

-- Initialize test data
-- db_a.table1
insert overwrite table db_a.table1 values('1','aliyun','emrA1'),('2','aliyun','dlfA1');

-- db_a.table2
insert overwrite table db_a.table2 values('1','aliyun','emrA2'),('2','aliyun','dlfA2');

-- db_b.table1
insert overwrite table db_b.table1 values('1','aliyun','emrB1'),('2','aliyun','dlfB1');

Create the RAM users required for each role.
- super administrator: Create a RAM user named dlf_data_admin.
- business A data administrator: Create a RAM user named dlf_dba_admin.
- business A data developer: Create a RAM user named dlf_dba_dev.
- business A data analyst: Create a RAM user named dlf_dba_analyst.
Enable access control.

To enable access control for the EMR cluster, complete the following steps:
1. Enable access control in the EMR cluster. For more information, see DLF-Auth.
2. Enable access control for the DLF Catalog in DLF. For more information, see Configure permissions.
  
  Note
  For production use, we recommend that you also enable LDAP authentication to ensure that user identities are properly validated. This tutorial does not enable LDAP authentication for simplicity. As a result, you are not prompted for a password when connecting with Beeline.
Configure permissions for the super administrator to manage permissions in the Data Lake Formation console and access all data.
1. Go to the Data Lake Formation console.
2. In the left-side navigation pane, choose Data Permission > Roles.
3. Add the dlf_data_admin user to the Admin role. The dlf_data_admin user is now an administrator who can manage all data permissions in DLF and configure them for any user.
4. To allow the dlf_data_admin user to configure data permissions for RAM users in the Data Lake Formation console, you must also attach the AliyunDLFFullAccess and AliyunRAMReadOnlyAccess policies to this user in the RAM console.
5. Log on to the EMR cluster, connect to Hive as the dlf_data_admin user, and run HiveQL statements.
```
beeline -u jdbc:hive2://<primary_node_name>:10000 -n dlf_data_admin
```
```
select * from db_a.table1;
select * from db_b.table1;
```
  The queries succeed, which indicates that the dlf_data_admin user has access to all databases and tables.
Configure permissions for the business A data administrator to grant permissions on the db_a database in the Data Lake Formation console and access all data in that database.
1. Switch to the dlf_data_admin RAM user in the Alibaba Cloud console. Use this account for the following authorization steps.
2. Go to the Data Lake Formation console.
3. In the left-side navigation pane, choose Data Permission > Data Permissions, and then click Add Permission.
4. Configure the following parameters:
  - Principal Type: RAM User/Role.
  - Choose Principal: dlf_dba_admin.
  - Resources: Resource Authorization.
  - Select Catalog: catalog_test.
  - Select Database: db_a.
  - Permissions:
    - Database - Data Permission: All.
    - Database - Granted Permission: All.
    - All objects in the database - Data Permission: All.
    - All objects in the database - Granted Permission: All.
5. Click OK to save the authorization.
6. To allow the dlf_dba_admin user to configure data permissions for other RAM users in the Data Lake Formation console, you must also attach the AliyunDLFFullAccess and AliyunRAMReadOnlyAccess policies to this user in the RAM console.
7. Log on to the EMR cluster, connect to Hive as the dlf_dba_admin user, and run HiveQL statements.
```
beeline -u jdbc:hive2://<primary_node_name>:10000 -n dlf_dba_admin
```
```
select * from db_a.table1;
select * from db_b.table1;
```
  The first query succeeds because the user has permissions for all resources in the db_a database.
  
  The second query fails because the user does not have permissions for resources in the db_b database.
Configure permissions for the business A data developer to query and modify data in the db_a database.
1. Switch to the dlf_dba_admin RAM user in the Alibaba Cloud console. Use this account to grant data permissions to other users.
2. Go to the Data Lake Formation console.
3. In the left-side navigation pane, choose Data Permission > Data Permissions, and then click Add Permission.
4. Configure the following parameters:
  - Principal Type: RAM User/Role.
  - Choose Principal: dlf_dba_dev.
  - Resources: Resource Authorization.
  - Select Catalog: catalog_test.
  - Select Database: db_a.
  - Permissions:
    - Database - Data Permission: All.
    - Database - Granted Permission: None.
    - All objects in the database - Data Permission: All.
    - All objects in the database - Granted Permission: All.
5. Click OK to save the authorization.
6. Log on to the EMR cluster, connect to Hive as the dlf_dba_dev user, and run HiveQL statements.
```
beeline -u jdbc:hive2://<primary_node_name>:10000 -n dlf_dba_dev
```
```
select * from db_a.table1;
insert into table db_a.table1 values('3','aliyun','emrA1'),('4','aliyun','dlfA1');

select * from db_b.table1;
insert into table db_b.table1 values('3','aliyun','emrA1'),('4','aliyun','dlfA1');
```
  The first and second SQL statements succeed, confirming that the dlf_dba_dev user has query and modify permissions on the db_a database and all of its resources.
  
  The third and fourth SQL statements fail because the user dlf_dba_dev does not have permissions to query or modify the db_b database.
Set permissions for the Business A data analyst to access table1(col1, col2) in the db_a database.
1. Log on to the Alibaba Cloud website as the dlf_dba_admin RAM user. Use this account to grant data permissions to other users.
2. Log on to the Data Lake Formation console.
3. In the navigation pane on the left, choose Data Permission > Data Permissions, and then click Add Permission.
4. Enter the following information:
  - Principal Type: RAM User/Role.
  - Choose Principal: dlf_dba_analyst.
  - Resources: Resource Authorization.
  - Resource Type: Column.
  - Select Data Catalog: catalog_test.
  - Select Database: db_a.
  - Select Table: table1.
  - Permissions:
    - Column-Data Permission: All.
5. Click OK to save the authorization information.
6. Log on to the EMR cluster, connect to Hive as dlf_dba_analyst, and run HiveSQL commands.
```
beeline -u jdbc:hive2://<name of the primary node>:10000 -n dlf_dba_analyst
```
```
select * from db_a.table1;
select col1,col2 from db_a.table1;
insert into table db_a.table1 values('5','aliyun','emrA1'),('6','aliyun','dlfA1');
select * from db_b.table1;
```
  The second SQL statement succeeds because the dlf_dba_analyst user has permission to query columns col1 and col2 in the db_a.table1 table.
  
  The first SQL statement fails because the dlf_dba_analyst user does not have permission to query column col3 in the db_a.table1 table.
  
  The third SQL statement fails because the dlf_dba_analyst user does not have permission to modify data in the db_a.table1 table.
  
  The fourth SQL statement fails because the dlf_dba_analyst user does not have permission to query data in the db_b database.

Summary

The preceding example demonstrates how to use DLF data permissions in a typical business scenario. You can configure permissions as needed to grant different users the appropriate level of access to various resources.