Write data to MaxCompute by using a custom RAM role - Simple Log Service

When a data shipping job of the new version is running to ship data to MaxCompute, the data that is read from a Logstore must be written to a MaxCompute table. To meet the requirement, you can authorize the data shipping job to assume a custom Resource Access Management (RAM) role. This topic describes how to authorize a MaxCompute data shipping job of the new version to ship data by using a custom RAM role.

Prerequisites

If you use a RAM user, make sure that the RAM user has the permissions to manage RAM roles.
A MaxCompute project is added to the required DataWorks workspace as the data source. For more information, see Add a MaxCompute data source.

Ship data within an Alibaba Cloud account

After you authorize a RAM role to write data to MaxCompute, a MaxCompute data shipping job can assume the RAM role to write the data of a Logstore to a MaxCompute table. To complete the authorization, you must add the RAM role as a workspace member.

Procedure

Create a RAM role. In this example, a role named MaxComputeShipRole is created. For more information, see Create a RAM role for a trusted Alibaba Cloud service.
Important
- When you create a RAM role, you must select Trusted Entity Type as Alibaba Cloud Service and select Trusted Service as Log Service.
- Check the trust policy of the role as follows. The Service content must at least include "log.aliyuncs.com".
```
{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "log.aliyuncs.com"
        ]
      }
    }
  ],
  "Version": "1"
}
```
Modify the trust policy of the RAM role.
1. Log on to the RAM console.
2. In the left-side navigation pane, choose Identities > Roles.
3. In the role list, find the RAM role and click the role name.
4. On the Trust Policy tab, click Edit Trust Policy.
5. Replace the existing script in the code editor with the following policy document and click Save trust policy document:
```
{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "log.aliyuncs.com",
          "dataworks.aliyuncs.com"
        ]
      }
    }
  ],
  "Version": "1"
}
```

Add the RAM role as a workspace member.

You can use the GUI or CLI to authorize the RAM role to write data to MaxCompute.

Use the GUI

Log on to the DataWorks console.
In the top navigation bar, select a region.
In the left-side navigation pane, click Workspace. On the Workspaces page, find the workspace that you want to manage and click Manage in the Actions column.
On the Workspace Members tab of the Workspace page, click Add Members.
In the Add Members dialog box, select the current logon account and the RAM role, and then add the logon account and the RAM role as prompted.
In the Batch Assign Roles section, select Development. For more information, see Grant permissions to a RAM user.
Authorize the RAM role to manage a MaxCompute table.
1. Log on to the MaxCompute console. In the top navigation bar, select a region.
2. In the left-side navigation pane, choose Workspace > Projects. On the Projects page, find the project that you want to manage and click Manage in the Actions column.
3. On the MaxCompute project management page, click the Role Permissions tab.
  If the following error is reported, perform the following operations in the role list: Find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select the current logon account and add the logon account as prompted.
4. In the role list, find the role_project_admin role and click Manage Members in the Actions column.
5. In the Manage Members dialog box, select the current logon account and the RAM role, and then add the logon account and the RAM role as prompted.
6. In the role list, find the role_project_admin role and click Edit Role in the Actions column.
7. On the Table tab of the Edit Role dialog box, find the MaxCompute table that you want to manage and select Describe, Alter, and Update.

Important

The preceding authorization procedure takes effect only on the specified MaxCompute table. If you want to authorize a RAM role to manage all tables in the current MaxCompute project, you can grant the permissions of the admin role to the current logon account and the RAM role. In the role list, find the admin role and click Manage Members in the Actions column. In the Manage Members dialog box, select the current logon account and the RAM role, and then add the logon account and the RAM role as prompted.

Use the CLI

Log on to the DataWorks console.
In the top navigation bar, select a region.
In the left-side navigation pane, click Workspace.
On the Workspaces page, find the workspace that you want to manage, move the pointer over Shortcuts, and then click Data Development.
Create a workflow.
1. On the Scheduled Workflow page, choose Create > Create Workflow.
2. In the Create Workflow dialog box, configure the Workflow Name parameter and click Create.
Create a node.
1. On the Scheduled Workflow page, choose Create > Create Node > MaxCompute > ODPS SQL.
2. In the Create Node dialog box, configure the Name and Path parameters, and click Confirm.
  You must set the Path parameter to the workflow that you created in the previous step.

In the code editor of the node, run the required commands to complete the authorization. The following table describes the commands.

MaxCompute授权

Command	Description
`USE project-name;`	Specifies a MaxCompute project. The MaxCompute project must be the same as the MaxCompute project that you specified when you created your data shipping job of the new version. For more information, see Create a data shipping job of the new version to ship data to MaxCompute.
ADD USER RAM$****.aliyunid.com:`role/maxcomputeshiprole`;	Adds the RAM role as a user to the MaxCompute project. ****.aliyunid.com specifies the Alibaba Cloud account to which the MaxCompute project belongs. You can run the `list users;` command to view the Alibaba Cloud account. maxcomputeshiprole specifies the name of the custom RAM role. The name must be in lowercase.
GRANT CreateInstance ON PROJECT project-name TO USER RAM$*****:`role/maxcomputeshiprole`;	Authorizes the RAM role to call the CreateInstance operation to create an instance in a MaxCompute project named project-name. `project-name` specifies the name of the MaxCompute project.
GRANT DESCRIBE, ALTER, UPDATE ON TABLE table-name to user RAM$****.aliyunid.com:`role/maxcomputeshiprole`;	Authorizes the RAM role to view, modify, and update the specified MaxCompute table. `table-name` specifies the name of the MaxCompute table. Note The authorization procedure applies only on the specified MaxCompute table. If you want to authorize the RAM role to manage all tables in the current MaxCompute project, run the GRANT admin to user RAM$****.aliyunid.com:`role/maxcomputeshiprole`; command to complete the authorization.
SHOW GRANTS FOR `RAM$****.aliyunid.com:role/maxcomputeshiprole`;	Checks whether the authorization is successful.

If information similar to the following code is returned, the authorization is successful:

Authorization Type: ACL
[user/RAM$****.aliyunid.com:role/maxcomputeshiprole]
A       projects/default_project_****: CreateInstance
A       projects/default_project_****/tables/****: Describe | Alter | Update

The following error messages may appear during the authorization process:

If the FAILED: mismatched input error message appears, the RAM user does not have the permissions to execute statements such as ADD USER. For more information, see What do I do if the error message FAILED: mismatched input appears?
If the FAILED: ODPS-0130013:Authorization exception - Authorization Failed [4003], You have NO privilege to do the PROJECT SECURITY OPERATION for {acs:odps:*:projects/xxxxxx/authorization/users}. Context ID:1111-11111-1111-1111-11111. error message appears, the user does not have the permissions to manage MaxCompute projects or grant permissions to other users. For more information, see What do I do if the system reports an error that a user does not have the required permissions?

What to do next

After you configure the settings, you can assign the custom RAM role to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, set the Write Permissions on MaxCompute parameter to Custom Role and specify the Alibaba Cloud Resource Name (ARN) of the RAM role. In this example, the ARN acs:ram::10**12:role/maxcomputeshiprole is specified. For more information, see Create a data shipping job of the new version to ship data to MaxCompute. MaxCompute授权

Ship data across Alibaba Cloud accounts

If Simple Log Service is activated for Alibaba Cloud Account A and MaxCompute is activated for Alibaba Cloud Account B, you must authorize the RAM role of Alibaba Cloud Account B to write data to MaxCompute. In this example, the MaxComputeShipRole role is authorized to write data to MaxCompute. After the authorization is complete, a MaxCompute data shipping job of the new version can assume the RAM role to write the data of a Logstore to a MaxCompute table.

Procedure

Modify the trust policy of the RAM role that belongs to Alibaba Cloud Account B.
1. Log on to the RAM console by using Alibaba Cloud Account B.
2. Create a RAM role. In this example, a role named MaxComputeShipRole is created. For more information, see Create a RAM role for a trusted Alibaba Cloud service.
  Important
  - When you create a RAM role, you must select Trusted Entity Type as Alibaba Cloud Service and select Trusted Service as Log Service.
  - Check the trust policy of the role as follows. The Service content must at least include "log.aliyuncs.com".
    { "Statement": [ { "Action": "sts:AssumeRole", "Effect": "Allow", "Principal": { "Service": [ "log.aliyuncs.com" ] } } ], "Version": "1" }
3. Modify the trust policy of the RAM role.
  1. In the left-side navigation pane, choose Identities > Roles.
  2. In the role list, find the RAM role and click the role name.
  3. On the Trust Policy tab, click Edit Trust Policy. Add {ID of Alibaba Cloud Account A}@log.aliyuncs.com and dataworks.aliyuncs.com to the Service element. Replace {ID of Alibaba Cloud Account A} with the actual ID. You can view the ID of your Alibaba Cloud account in the Account Management console.
    Note
    The following policy allows Alibaba Cloud Account A to obtain a temporary Security Token Service (STS) token to manage the cloud resources of Alibaba Cloud Account B:
```
{
    "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "log.aliyuncs.com",
                    "dataworks.aliyuncs.com",
                    "{ID of Alibaba Cloud Account A}@log.aliyuncs.com"
                ]
            }
        }
    ],
    "Version": "1"
}
```
  4. After you confirm the settings, click Save trust policy document.
Add the RAM role as a workspace member.
You can use the GUI or CLI to grant permissions to the RAM role. For more information, see Use the GUI to grant permissions to the RAM role or Use the CLI to grant permissions to the RAM role.

What to do next

After you configure the settings, you can assign the custom RAM role of Alibaba Cloud Account B to a MaxCompute data shipping job of the new version to ship data to the specified MaxCompute table. When you create the data shipping job, set the Write Permissions on MaxCompute parameter to Custom Role and specify the ARN of the RAM role. In this example, the ARN acs:ram::11**13:role/maxcomputeshiprole is specified. For more information, see Create a data shipping job of the new version to ship data to MaxCompute. MaxCompute授权