How to grant data access permissions - Simple Log Service

Data transformation involves two types of authorization:

Permissions to perform data transformation operations, such as creating, deleting, modifying, and viewing data transformation jobs, and previewing data in the console or using a software development kit (SDK).
Permissions for a data transformation job to access data. This includes reading data from a source Logstore and writing the transformation results to a destination Logstore.

Grant a RAM user permissions to perform data transformation operations

Data transformation operations include creating, deleting, modifying, and viewing data transformation jobs, and previewing data. Before you grant permissions, ensure that you have created a RAM user. You can grant permissions in one of the following two ways:

Use a system policy: This method is simple to configure but provides coarse-grained control over a wide range of permissions.
Use a custom policy: This method provides fine-grained control over operations for specific projects or Logstores. It is suitable for environments that have strict permission requirements.

Use a system policy

Log on to the Resource Access Management (RAM) console using an Alibaba Cloud account or as a RAM administrator.
For more information, see Manage RAM user permissions. Attach the following two system policies to the RAM user:
- AliyunRAMFullAccess
- AliyunLogFullAccess
After you save the configuration, the RAM user can perform the following operations:
- Create, delete, modify, and view data transformation jobs.
- Read data from source Logstores to preview jobs.

Use a custom policy

Log on to the RAM console using an Alibaba Cloud account or as a RAM administrator.

Create a custom policy in script mode. For this example, name the policy log-etl-operator-policy. On the Script Editor tab, replace the existing content with the following template. In the template, replace the project name and Logstore name with your actual values.

{
    "Version":"1",
    "Statement":[
        {
            "Effect":"Allow",
            "Action":[
                "log:CreateLogStore",
                "log:CreateIndex",
                "log:UpdateIndex",
                "log:Get*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/logstore/internal-etl-log"
        },
        {
            "Action":[
                "log:List*"
            ],
            "Resource":"acs:log:*:*:project/*",
            "Effect":"Allow"
        },
        {
            "Action":[
                "log:Get*",
                "log:List*"
            ],
            "Resource":[
                "acs:log:*:*:project/your-project-name/logstore/your-logstore-name"
            ],
            "Effect":"Allow"
        },
        {
            "Effect":"Allow",
            "Action":[
                "log:GetDashboard",
                "log:CreateDashboard",
                "log:UpdateDashboard"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/dashboard/internal-etl-insight*"
        },
        {
            "Effect":"Allow",
            "Action":"log:CreateDashboard",
            "Resource":"acs:log:*:*:project/your-project-name/dashboard/*"
        },
        {
            "Effect":"Allow",
            "Action":[
                "log:*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/job/*"
        },{
            "Effect":"Allow",
            "Action":[
                "log:*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/etl/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ram:PassRole",
                "ram:GetRole",
                "ram:ListRoles"
            ],
            "Resource": "*"
         }
    ]
}

Attach the custom policy that you created to the RAM user.

Authorize a data transformation job to access Logstore data

Write to a Logstore in the same account

This scenario applies if the source Logstore and the destination Logstore belong to the same Alibaba Cloud account. You can grant permissions using a default system role or a custom role.

Use a default role

When you create a data transformation job, you can select the default role AliyunLogETLRole. The AliyunLogETLRole system role has the following built-in permissions:

Read data from the source Logstore.
Write transformation results to a destination Logstore in the same account.

Use a custom role

Step 1: Create a RAM role and configure a trust policy

Trusted Entity Type: Alibaba Cloud Service
Trusted Entity Name: Simple Log Service
After the role is created, verify its trust policy. The Service parameter must contain at least "log.aliyuncs.com".

Step 2: Grant the RAM role read-only access to the source Logstore

Create a custom policy. For example, name it log-etl-source-reader-policy. This policy allows the custom role to read data from a specific source Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.

Exact authorization

For example, the source project is named log-project-prod, and the source Logstore is named access_log. Replace these names with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Wildcard authorization

Example scenario: Replace the following names with your actual values.

The source projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
The source Logstores are named app_a_log, app_b_log, app_c_log, and so on. In this case, use wildcard authorization.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
    "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

For more information, see Grant permissions to a RAM role. Attach this policy to the custom RAM role that you created in Step 1.

Step 3: Grant the RAM role permissions to write data to the destination Logstore

Create a custom policy. For example, name it log-etl-target-writer-policy. This policy allows the custom role to write transformation results to the destination Logstore.
On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.
Exact authorization
For example, the destination project is named log-project-prod, and the destination Logstore is named access_log_output. Replace these names with your actual values.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*"
      ],
      "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Wildcard authorization
Example scenario: Replace the following names with your actual values.
- The destination projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
- The destination Logstores are named app_a_log_output, app_b_log_output, app_c_log_output, and so on. In this case, use wildcard authorization.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*",
        "log:BatchPost*"
      ],
       "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
      "Effect": "Allow"
    }
  ]
}
```
For more information, see Grant permissions to a RAM role. Attach this policy to the custom RAM role that you created in Step 1.

Write to a Logstore in a different account

This scenario applies if the source Logstore and the destination Logstore belong to different Alibaba Cloud accounts. In this case, you can grant access to Logstore data only using a custom role.

Example scenario: You need to transform data from a source Logstore in Alibaba Cloud account A and save the results to a destination Logstore in Alibaba Cloud account B. To do this, you must log on to Alibaba Cloud account B and grant permissions as described in the following steps.

Procedure

Step 1: Create a RAM role and configure a trust policy

Create two RAM roles, RAM role A and RAM role B, in Account A and Account B, respectively. Account A is the account that owns the source Logstore, and Account B is the account that owns the destination Logstore.

Trusted Entity Type: Alibaba Cloud Service
Trusted Entity Name: Simple Log Service
After the roles are created, verify their trust policies. The Service parameter must contain at least "log.aliyuncs.com".

Step 2: Grant permissions to read data from the source Logstore

In Alibaba Cloud account A, which owns the source Logstore, create a custom policy. For example, name it log-etl-source-reader-policy. This policy allows the custom role to read data from a specific source Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.

Exact authorization

For example, the source project is named log-project-prod, and the source Logstore is named access_log. Replace these names with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Wildcard authorization

For example, the source projects are named log-project-dev-a, log-project-dev-b, and log-project-dev-c, and the source Logstores are named app_a_log, app_b_log, and app_c_log. In this case, use wildcard authorization. Replace these names with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
    "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Attach this policy to the custom RAM role A that you created in Step 1.

Step 3: Grant permissions to write data to the destination Logstore

In Alibaba Cloud account B, which owns the destination Logstore, create a custom policy. For example, name it log-etl-target-writer-policy. This policy allows the custom role to write transformation results to the destination Logstore.
On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.
Exact authorization
For example, the destination project is named log-project-prod, and the destination Logstore is named access_log_output. Replace these names with your actual values.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*"
      ],
      "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Wildcard authorization
Example scenario: Replace the following names with your actual values.
- The destination projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
- The destination Logstores are named app_a_log_output, app_b_log_output, app_c_log_output, and so on. In this case, use wildcard authorization.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*",
        "log:BatchPost*"
      ],
       "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Attach this policy to the custom RAM role B that you created in Step 1.

Step 4: Configure the trust policy

In Alibaba Cloud account B, which owns the destination Logstore:

Log on to the RAM console.
In the navigation pane on the left, choose Identity Management > Roles.
In the RAM role list, click the destination RAM role.

On the Trust Policy Management tab, click Edit Trust Policy:

{
  "Version": "1",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ID-of-account-A-that-owns-the-source-Logstore@log.aliyuncs.com"
        ]
      }
    }
  ]
}

Note:

Replace "ID-of-account-A-that-owns-the-source-Logstore@log.aliyuncs.com" with the ID of Account A, such as 1234567890123***@log.aliyuncs.com.
This policy indicates that Account A can obtain temporary credentials from Simple Log Service to assume this role in Account B. This allows Account A to access the resources that are configured in Account B, such as the destination Logstore.
In the role details, record the Alibaba Cloud Resource Name (ARN) of the role. You will need this ARN when you create the data transformation job in Account A.