Intra-account log transfer with custom roles - Simple Log Service

RAM users can use custom roles to transfer log data within the same Alibaba Cloud account during data transformation.

Prerequisites

Source Logstore, destination Logstore, and Project are created, and you have their names. Manage Logstores, Manage Projects.
A RAM user with data transformation permissions is created. Grant RAM users permissions to perform data transformation.

Background information

When a RAM user creates a data transformation job within a single account, RAM role A requires read permissions on the source Logstore and RAM role B requires write permissions on the destination Logstore. The following figure shows the authorization model.

Step 1: Create role A

Log on to the RAM console with your Alibaba Cloud account or as a RAM administrator.

Create RAM role A.

For more information, see Create a RAM role for a trusted Alibaba Cloud service. Configure the key parameters as described in the following table.

Key parameter	Description
Select type of trusted entity	Select Alibaba Cloud Service.
Role Type	Select Normal Service Role.
Role Name	Enter a role name, such as role-A.
Select Trusted Service	Select Simple Log Service.

Step 2: Grant read permissions

Log on to the RAM console with your Alibaba Cloud account or as a RAM administrator.

In the script editor mode, create a custom policy that allows you to read data from the source Logstore. For example, name the policy ori_read.

For more information, see Create a custom policy. Configure the key parameters as described in the following table.

Key parameter

Description

Name

Enter a name for the custom policy. For example, enter ori_read.

Policy Document

Replace the existing script in the editor with the following content.

In this example, the source project is log-project-prod and the source Logstore is access_log. Replace these values based on your actual configuration.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Grant RAM role A read permissions on the source Logstore.

For more information, see Manage permissions for a RAM role. Configure the key parameters as described in the following table.

Key parameter	Description
Grant Permission On	Select Account Level. The permissions take effect within the current Alibaba Cloud account.
Principal	Select role-A, which you created in Step 1: Create RAM role A.
Custom Policy	Select ori_read.

Obtain the Alibaba Cloud Resource Name (ARN) of the RAM role.

On the details page for RAM role A, find the ARN in the Basic Information section. Example: acs:ram::1379******44:role/role-a.

Step 3: Create role B

Create RAM role B.

For more information, see Create a RAM role for a trusted Alibaba Cloud service. Configure the key parameters as described in the following table.

Key parameter	Description
Select type of trusted entity	Select Alibaba Cloud Service.
Role Type	Select Normal Service Role.
Role Name	Enter a role name, such as role-B.
Select Trusted Service	Select Simple Log Service.

Step 4: Grant write permissions

Create a custom policy in script editing mode to write data transformation results to the destination LogStore. For example, you can create a policy named write.

For more information, see Create a custom policy. Configure the key parameters as described in the following table.

Key parameter	Description
Name	Enter a name for the custom policy. For example, enter write.
Policy Document	Replace the existing script in the editor with the following content. In this example, the destination project is `log-project-prod` and the destination Logstore is `access_log_output`. Replace these values based on your actual configuration. `{ "Version": "1", "Statement": [ { "Action": [ "log:Post", "log:BatchPost" ], "Resource": "acs:log:::project/log-project-prod/logstore/access_log_output", "Effect": "Allow" } ] }`

Grant RAM role B write permissions on the destination Logstore.

For more information, see Manage permissions for a RAM role. Configure the key parameters as described in the following table.

Key parameter	Description
Grant Permission On	Select Account Level. The permissions take effect within the current Alibaba Cloud account.
Principal	Select role-B, which you created in Step 3: Create RAM role B.
Custom Policy	Select write.

Obtain the Alibaba Cloud Resource Name (ARN) of the RAM role.

On the details page for RAM role B, find the ARN in the Basic Information section. Example: acs:ram::1379******44:role/role-b.

Create a custom policy in script editor mode. This policy allows you to write data transformation results to the destination Logstore. For example, name the policy write.

For more information, see Create a custom policy. Configure the key parameters as described in the following table.

Key parameter	Description
Name	Enter a name for the custom policy. For example, enter write.
Policy Document	Replace the existing script in the editor with the following content. In this example, the destination project is log-project-prod and the destination Logstore is access_log_output. Replace these values based on your actual configuration. `{ "Version": "1", "Statement": [ { "Action": [ "log:Post", "log:BatchPost" ], "Resource": "acs:log:::project/log-project-prod/logstore/access_log_output", "Effect": "Allow" } ] }`

Grant Role B write permissions on the target Logstore.

For more information, see Manage the permissions of a RAM role. Configure the key parameters as described in the following table.

Key parameter	Description
Grant Permission On	Select Account Level. The permissions take effect within the current Alibaba Cloud account.
Principal	Select role-B, which you created in Step 3: Create role B.
Custom Policy	Select write.

Obtain the RAM role ARN.

On the details page for RAM role B, find the ARN in the Basic Information section. Example: acs:ram::1379******44:role/role-b.

Step 5: Create data transformation job

Log on to the Log Service console as the RAM user.
Go to the data transformation page.
1. In the Projects section, click the project you want.
2. On the Log Storage > Logstores tab, click the logstore you want.
3. On the Query and Analysis page, click Data Transformation.
In the upper-right corner of the page, select a time range for the data.

Make sure that log data appears on the Raw Logs tab.
Enter a data transformation statement in the editor.

Data transformation syntax.
Preview the data.
1. Click Quick.
  
  Simple Log Service supports quick preview and advanced preview. Preview and debug.
2. Click Preview Data.
  Review the preview results.
  - If the transformation fails due to incorrect statements or misconfigured permissions, follow the on-screen instructions to resolve the issue.
  - If the transformation result is correct, proceed to the next step.

Create a data transformation job.

Click Save as Transformation Job.

In the Create Data Transformation Job panel, configure the parameters and click OK.

For information about other parameters, see Data transformation quick start. The following table describes the key parameters.

Key parameter	Description
Authorization Method	Select Custom Role.
Role ARN	Enter the ARN of RAM role A. Example: `acs:ram::1379******44:role/role-a`.
Authorization Method for Storage Destination	Select Custom Role.
Role ARN	Enter the ARN of RAM role B. Example: `acs:ram::1379******44:role/role-b`.

After the data transformation job starts running, the intra-account data transfer is configured. For more information, see Manage data transformation jobs.