Configure a data source using RAM role-based authorization - DataWorks

This topic uses an OSS data source as an example to describe how to configure a data source using RAM role-based authorization. This method improves the security of your data in the cloud.

Prerequisites

To log on as a Resource Access Management (RAM) user to complete the steps in this topic, you must first grant the AliyunDataWorksFullAccess and AliyunRAMFullAccess policies to the RAM user. For more information, see Manage RAM user permissions.

Note

If you are using an Alibaba Cloud account, you can skip this prerequisite.

The following figure shows how to add permissions.

Background information

Data sources are the foundation for data synchronization tasks and affect the security of your company's data in the cloud. DataWorks lets you use the more secure RAM role-based authorization mode to configure and access data sources, such as OSS, AnalyticDB for MySQL, LogHub, Tablestore, and Hologres. This mode improves the security of your cloud data and helps prevent issues such as data source misuse and AccessKey leaks.

Data source access modes include RAM role-based authorization and AccessKey mode. This topic describes how to configure a data source using RAM role-based authorization. You can choose the mode that fits your business needs. The principles of the AccessKey mode and the RAM role-based authorization mode are as follows:

AccessKey mode
In the less secure AccessKey mode, which uses an AccessKey ID and an AccessKey secret, you can complete the configuration by entering the AccessKey of an Alibaba Cloud account or a RAM user.
For example, to configure an OSS data source, you can enter the AccessKey of an account with permission to access a specific OSS bucket on the Configure Data Source page.
When you execute or schedule a sync task, the task uses this AccessKey to access OSS and read or write data.
Note
In AccessKey mode, if an AccessKey is leaked, its associated OSS data is also exposed.
RAM role-based authorization
The RAM role-based authorization mode provides a more secure way to access data sources. It does not require an AccessKey, which effectively avoids the risk of AccessKey leaks.
In RAM role-based authorization mode, you only need to grant a role with OSS access permissions to the DataWorks service account. This allows access to the OSS data source without an AccessKey.
This mode also meets the needs of enterprise users. It lets you assign roles with different permission scopes to different data sources for more professional permission management.

Process overview

To allow RAM users to perform the entire process instead of the root account, this topic describes the conditions for a RAM user to perform each step. The process of using RAM role-based authorization is as follows.

An Alibaba Cloud account or a RAM user that is granted the AliyunRAMFullAccess policy goes to the RAM console to define the role to be assumed and the policy to be granted.
- Role to be assumed: Create a custom role for the DataWorks service account to assume. After the DataWorks service account assumes the role, it can access the OSS data source within the scope of the permissions that are granted to the role.
- Policy to be granted: Create a policy that includes the `PassRole` permission. This policy grants a user the permission to use a specific role to create a data source or run a sync task.
An Alibaba Cloud account or a RAM user that is granted the AliyunRAMFullAccess policy goes to the RAM console to grant the RAM users who will perform Step 4 and Step 5 the permission to use the role.
Note
If a RAM user who has not been granted this permission creates a data source using the RAM role-based authorization mode, all sync tasks that are configured with that data source fail.
The data source creator goes to DataWorks Data Integration to create a data source in RAM role-based authorization mode. This allows the DataWorks service account to assume a role to access the OSS data source when a sync task runs.
Note
This step is only available if the data source creator was authorized in Step 3.
The sync task creator goes to Data Studio to create a sync task based on the configured data source.
The executor runs the data sync task in Data Studio or Operation Center.
Note
This step can be performed only after the task executor is authorized in Step 3.

Procedure

Create the role to be assumed.
Based on your security requirements, you can create different custom roles for different data sources. This topic uses the following scenario as an example to show how to create the role to be assumed.
Note
Only an Alibaba Cloud account or a RAM user with the AliyunRAMFullAccess permission can perform this step.
A company has 100 buckets that store all of its data. The big data team needs to use data from only one of these buckets. If the preset AliyunDataWorksAccessingOSSRole role is used, the big data team might be able to access the other 99 buckets, which poses a management risk.
Therefore, the Alibaba Cloud account owner can create a custom role named BigDataOssRole for the big data team. The owner can also restrict the use of this role to specific members of the big data team to manage permissions between teams.
1. Create a custom role.
  This example shows how to create a custom role named BigDataOssRole with Alibaba Cloud Account as the trusted entity. For more information, see Create a RAM role for a trusted Alibaba Cloud account.
2. Create a custom policy.
  This example shows how to create a policy that grants read and write permissions on a specific bucket. For more information about how to create a policy, see Create a custom policy. The policy content is as follows.
```
{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "oss:GetObject",
                "oss:ListObjects",
                "oss:GetObjectMetadata",
                "oss:GetObjectMeta",
                "oss:GetBucketAcl",
                "oss:GetBucketInfo",
                "oss:PutObject",
                "oss:DeleteObject",
                "oss:PutBucket"
            ],
            "Resource": [
                "acs:oss:*:*:bucket_name_1",
                "acs:oss:*:*:bucket_name_1/*"
            ]
        }
    ]
}
```
3. Grant the specified policy to the role.
  Modify the trust policy of the BigDataOSSRole role to attach the BigDataOSSRole role the policy that you created in the previous step. This allows users who are granted the BigDataOSSRole role to read data from the two specified buckets.
  Important
  You must perform this step. Otherwise, the role cannot be used.
  For more information about how to modify the trust policy of the role, see Modify the trust policy of a RAM role. The policy content is as follows.
```
{
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "di.dataworks.aliyuncs.com"
                ]
            }
        }
    ],
    "Version": "1"
}
```
Grant users permission to use the role.
After you confirm all the roles to be assumed, you must grant a policy that includes the `PassRole` permission to the relevant users. These users can then use the role to create data sources and run sync tasks. You can also configure the mapping between users and roles based on your scenario to specify which users can use which roles.
- Policy Template 1: You can create a policy using the following template. This template allows the grantee to use all roles that are related to DataWorks Data Integration. Grant this policy with caution.
```
{
    "Version": "1",
    "Statement": [
        {
            "Action": "ram:PassRole",
            "Resource": "*",
            "Effect": "Allow",
            "Condition": {
                "StringEquals": {
                    "acs:Service": "di.dataworks.aliyuncs.com"
                }
            }
        }
    ]
}
```
- Policy Template 2: Based on your security requirements, you can create a custom policy that includes the `PassRole` permission to configure the mapping between users and roles.
  Note
  Only an Alibaba Cloud account or a RAM user with the AliyunRAMFullAccess permission can perform this step.
  Scenario example: As described in the scenario for the role to be assumed, after the account administrator defines the BigDataOssRole role for the big data team, the administrator needs to specify that only relevant users can use this role. You can create a custom policy named BigDataOssRoleAllowUse and grant it to the relevant users.
  Create a policy named BigDataOssRoleAllowUse. For more information, see Create a custom policy. The policy script is as follows:
```
{
    "Version": "1",
    "Statement": [
        {
            "Action": "ram:PassRole",
            "Resource": "acs:ram::19122324****:role/BigDataOssRole",
            "Effect": "Allow",
            "Condition": {
                "StringEquals": {
                    "acs:Service": [
                        "oss.aliyuncs.com",
                        "di.dataworks.aliyuncs.com"
                    ]
                }
            }
        }
    ]
}
```
  Note
  Replace the UID in the policy (19122324****) with the UID of the current Alibaba Cloud account.
  The administrator grants the BigDataOssRoleAllowUse policy to the RAM users who are allowed to use the BigDataOssRole role. The RAM users who are granted the BigDataOssRoleAllowUse policy can then use the BigDataOssRole role to create data sources and run sync tasks. When you create a data source, you can use BigDataOssRole as the access identity for the data source.

Create the data source.

After the account administrator grants permissions to the data source creator, the creator can create the data source.

An Alibaba Cloud account or a RAM user with the AliyunDataWorksFullAccess policy creates and configures an OSS data source.

During configuration, set Access Mode to RAM role-based authorization. Configure the other parameters as described in this topic. If you are using a standard mode workspace, you can configure the Development or Production environment as needed.

Note

This topic uses an OSS data source as an example. The actual interface may vary depending on the data source you use. For more information about configuring an OSS data source, see Configure an OSS data source.

Parameter	Description
Data Source Name	The data source name must consist of letters, digits, and underscores (_). It cannot start with a digit or an underscore (_).
Data Source Description	A brief description of the data source. The description cannot exceed 80 characters.
Endpoint	The OSS Endpoint. The format is `http://oss.aliyuncs.com`. The OSS service Endpoint varies by region. You need to enter a different domain name when accessing different regions. Note The correct format for the Endpoint is `http://oss.aliyuncs.com`. Do not add the bucket name before `oss.aliyuncs.com` with a period, such as `http://xxx.oss.aliyuncs.com`. Although the connectivity test may pass with this format, the synchronization will fail.
Bucket	The OSS bucket information. A bucket is a container for storing objects. You can create one or more buckets. You can add one or more files to each bucket. In a data sync task, you can find the corresponding files in the bucket entered here. You cannot find files in buckets that are not added.
Access Mode	Select RAM role-based authorization. This mode uses Security Token Service (STS) authorization to allow the cloud product's service account to assume a role to access the data source, providing higher security.
Select Role	Select a RAM role from the Select Role drop-down list.
Region	Select a region from the Region drop-down list.

Test network connectivity.
In the Connection Configuration section, click Test Connectivity for the resource group.
A data sync task can use only one type of resource group. You must test the connectivity of the resource group to ensure that the data integration resource group that is used by the sync task can connect to the data source. Otherwise, the data sync task cannot run properly. To test multiple resource groups at the same time, you can select the resource groups and click Batch Test Connectivity. For more information, see Network connectivity solutions.
After the connectivity test is successful, click Complete.

Create a sync task.
After the data source is created, a developer can go to the Data Studio page to create a sync task based on this data source. For more information, see Configure a sync task.
Run the sync task.
The task executor runs the data sync task on the Data Studio or Operation Center page.
Note
When you run a task in Data Studio, make sure that the task executor has been granted the required permission in Step 2: Grant users permission to use the role. This prevents task failure.