Use custom roles to transfer data within the same Alibaba Cloud account - Simple Log Service

When you use a Resource Access Management (RAM) user to create a data transformation job, you can specify custom roles to transfer data within the same Alibaba Cloud account.

Prerequisites

A project is created. A source Logstore and a destination Logstore are created in the project. The names of the source and destination Logstores and the name of the project are obtained. For more information, see Manage a logstore and Manage projects.
A RAM user is created, and the RAM user is granted the required permissions to perform data transformation operations. For more information, see Grant a RAM user the permissions to manage a data transformation job.

Background information

In a scenario that involves only one Alibaba Cloud account, a RAM user is used to create a data transformation job. Role A must have the read permissions on the source Logstore, and Role B must have the write permissions on the destination Logstores. The following figure shows the requirements for authorization.

Step 1: Use your Alibaba Cloud account to create Role A

Log on to the RAM console by using your Alibaba Cloud account.

Create Role A.

For more information, see Create a RAM role for a trusted Alibaba Cloud service. The following table describes the key parameters.

Parameter	Description
Select Trusted Entity	Select Alibaba Cloud Service.
Role Type	Select Normal Service Role.
RAM Role Name	Enter a name for the role. In this example, enter role-A.
Select Trusted Service	Select Log Service.

Step 2: Use your Alibaba Cloud account to grant the read permissions to Role A

Log on to the RAM console by using your Alibaba Cloud account.

Create a custom policy on the JSON tab. The policy grants the permissions to read data from the source Logstore. In this example, create a policy named ori_read.

For more information, see Create a custom policy. The following table describes the key parameters.

Parameter

Description

Name

Enter a name for the custom policy. In this example, enter ori_read.

Policy document

Replace the content in the code editor with the following script.

In this example, the source project is named log-project-prod, and the source Logstore is named access_log. Replace the project and Logstore names based on your business requirements.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Grant the read permissions on the source Logstore to Role A.

For more information, see Grant permissions to a RAM role. The following table describes the key parameters.

Parameter	Description
Authorized Scope	Select Alibaba Cloud Account. The authorization takes effect within the current Alibaba Cloud account.
Principal	Select role-A. role-A is the name of Role A that is created in Step 1: Use your Alibaba Cloud account to create Role A.
Custom Policy	Select ori_read.

Obtain the Alibaba Cloud Resource Name (ARN) of Role A.
In the Basic Information section on the details page of Role A, obtain the ARN of the role. Example: acs:ram::1379******44:role/role-a.

Step 3: Use your Alibaba Cloud account to create Role B

Log on to the RAM console by using your Alibaba Cloud account.

Create Role B.

For more information, see Create a RAM role for a trusted Alibaba Cloud service. The following table describes the key parameters.

Parameter	Description
Select Trusted Entity	Select Alibaba Cloud Service.
Role Type	Select Normal Service Role.
RAM Role Name	Enter a name for the role. In this example, enter role-B.
Select Trusted Service	Select Log Service.

Step 4: Use your Alibaba Cloud account to grant the write permissions to Role B

Log on to the RAM console by using your Alibaba Cloud account.

Create a custom policy on the JSON tab. The policy grants the permissions to write data to the destination Logstores. In this example, create a policy named write.

For more information, see Create a custom policy. The following table describes the key parameters.

Parameter	Description
Name	Enter a name for the custom policy. In this example, enter write.
Policy document	Replace the content in the code editor with the following script. In this example, the destination project is named log-project-prod, and the destination Logstore is named access_log_output. Replace the project and Logstore names based on your business requirements. `{ "Version": "1", "Statement": [ { "Action": [ "log:Post", "log:BatchPost" ], "Resource": "acs:log:::project/log-project-prod/logstore/access_log_output", "Effect": "Allow" } ] }`

Grant the write permissions on the destination Logstore to Role B.

For more information, see Grant permissions to a RAM role. The following table describes the key parameters.

Parameter	Description
Authorized Scope	Select Alibaba Cloud Account. The authorization takes effect within the current Alibaba Cloud account.
Principal	Select role-B. role-B is the name of Role B that is created in Step 3: Use your Alibaba Cloud account to create Role B.
Custom Policy	Select write.

Obtain the ARN of Role B.
In the Basic Information section on the details page of Role B, obtain the ARN of the role. Example: acs:ram::1379******44:role/role-b.

Step 5: Use the RAM user to create a data transformation job

Log on to the Simple Log Service console by using the RAM user.
Go to the data transformation page.
1. In the Projects section, click the project you want.
2. On the Log Storage > Logstores tab, click the logstore you want.
3. On the query and analysis page, click Data Transformation.
In the upper-right corner of the page, specify a time range for the log data that you want to transform.
Make sure that log data exists on the Raw Logs tab.
In the code editor, enter a data transformation statement.
For more information, see Data transformation syntax.
Preview transformation results.
1. Click Quick.
  You can select Quick or Advanced. For more information, see Preview mode.
2. Click Preview Data.
  View the preview results.
  - If data fails to be transformed because the specified statement is invalid or the permissions that are granted are invalid, follow the on-screen instructions to troubleshoot the failure.
  - If the transformed data is returned as expected, go to the next step.

Create a data transformation job.

Click Save as Transformation Job.

In the Create Data Transformation Job panel, configure the parameters and click OK.

For more information about the parameters, see Get started with data transformation. The following table describes the key parameters.

Parameter	Description
Authorization Method	Click Custom Role.
Role ARN	Enter the ARN of Role A. Example: `acs:ram::1379******44:role/role-a`.
Authorization Method in Storage Destination	Click Custom Role.
Role ARN	Enter the ARN of Role B. Example: `acs:ram::1379******44:role/role-b`.

After the data transformation job is created and run, data can be transferred within the same Alibaba Cloud account. For more information, see Manage a data transformation job.