SLS data transformation job permissions and RAM authorization - Simple Log Service

Permissions to manage data transformation jobs (create, delete, modify, view) and preview data in the console or by using an SDK.
Data access permissions for transformation jobs to read from a source Logstore and write results to a destination Logstore.

Grant a RAM user permissions to perform data transformation operations

These operations include creating, deleting, modifying, and viewing transformation jobs, and previewing data. Make sure you have created a RAM user. You can grant permissions in either of the following ways:

Use a system policy: Simple to configure but grants broad, coarse-grained permissions.
Use a custom policy: Grants fine-grained control over specific projects or Logstores. Recommended for strict permission requirements.

Use a system policy

Log on to the Resource Access Management (RAM) console using an Alibaba Cloud account or as a RAM administrator.
Attach the following system policies to the RAM user (Manage RAM user permissions):
- AliyunRAMFullAccess
- AliyunLogFullAccess
After you save the configuration, the RAM user can:
- Create, delete, modify, and view data transformation jobs.
- Read data from a source Logstore to preview jobs.

Use a custom policy

Log on to the RAM console using an Alibaba Cloud account or as a RAM administrator.

Create a custom policy in script mode. For example, name it log-etl-operator-policy. On the Script Editor tab, replace the content with the following template. Replace the project and Logstore names with your actual values.

{
    "Version":"1",
    "Statement":[
        {
            "Effect":"Allow",
            "Action":[
                "log:CreateLogStore",
                "log:CreateIndex",
                "log:UpdateIndex",
                "log:Get*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/logstore/internal-etl-log"
        },
        {
            "Action":[
                "log:List*"
            ],
            "Resource":"acs:log:*:*:project/*",
            "Effect":"Allow"
        },
        {
            "Action":[
                "log:Get*",
                "log:List*"
            ],
            "Resource":[
                "acs:log:*:*:project/your-project-name/logstore/your-logstore-name"
            ],
            "Effect":"Allow"
        },
        {
            "Effect":"Allow",
            "Action":[
                "log:GetDashboard",
                "log:CreateDashboard",
                "log:UpdateDashboard"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/dashboard/internal-etl-insight*"
        },
        {
            "Effect":"Allow",
            "Action":"log:CreateDashboard",
            "Resource":"acs:log:*:*:project/your-project-name/dashboard/*"
        },
        {
            "Effect":"Allow",
            "Action":[
                "log:*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/job/*"
        },{
            "Effect":"Allow",
            "Action":[
                "log:*"
            ],
            "Resource":"acs:log:*:*:project/your-project-name/etl/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ram:PassRole",
                "ram:GetRole",
                "ram:ListRoles"
            ],
            "Resource": "*"
         }
    ]
}

Attach the custom policy that you created to the RAM user.

Authorize a data transformation job to access Logstore data

Write to a Logstore in the same account

When the source and destination Logstores belong to the same Alibaba Cloud account, you can grant permissions by using a default system role or a custom role.

Use a default role

When you create a data transformation job, select the default role AliyunLogETLRole. The AliyunLogETLRole system role has the following built-in permissions:

Read data from the source Logstore.
Write transformation results to a destination Logstore in the same account.

Use a custom role

Step 1: Create a RAM role and configure a trust policy

Trusted Entity Type: Alibaba Cloud Service
Trusted Entity Name: Simple Log Service
After you create the role, verify that the Service field in the trust policy contains at least "log.aliyuncs.com".

Step 2: Grant the RAM role read-only access to the source Logstore

Create a custom policy. For example, name it log-etl-source-reader-policy. This policy grants read access to a specific source Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.

Exact authorization

In this example, the source project is log-project-prod and the source Logstore is access_log. Replace these with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Wildcard authorization

Example: Replace the values with your actual requirements.

The source projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
The source Logstores are named app_a_log, app_b_log, app_c_log, and so on.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
    "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Attach this policy to the RAM role that you created in Step 1 (Grant permissions to a RAM role).

Step 3: Grant the RAM role permissions to write data to the destination Logstore

Create a custom policy. For example, name it log-etl-target-writer-policy. This policy grants write access to the destination Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.
Exact authorization

In this example, the destination project is log-project-prod and the destination Logstore is access_log_output. Replace these with your actual values.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*"
      ],
      "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Wildcard authorization
Example: Replace the values with your actual requirements.
- The destination projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
- The destination Logstores are named app_a_log_output, app_b_log_output, app_c_log_output, and so on.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*",
        "log:BatchPost*"
      ],
       "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Attach this policy to the RAM role that you created in Step 1 (Grant permissions to a RAM role).

Write to a Logstore in a different account

When the source and destination Logstores belong to different Alibaba Cloud accounts, you must use a custom role to grant access.

Example: To transform data from a source Logstore in Account A and write results to a destination Logstore in Account B, log on to Account B and grant permissions as follows.

Procedure

Step 1: Create a RAM role and configure a trust policy

Create RAM role A in Account A (source Logstore owner) and RAM role B in Account B (destination Logstore owner).

Trusted Entity Type: Alibaba Cloud Service
Trusted Entity Name: Simple Log Service
After you create the roles, verify that the Service field in each trust policy contains at least "log.aliyuncs.com".

Step 2: Grant permissions to read data from the source Logstore

In Account A (source Logstore owner), create a custom policy. For example, name it log-etl-source-reader-policy. This policy grants read access to a specific source Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.

Exact authorization

In this example, the source project is log-project-prod and the source Logstore is access_log. Replace these with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-prod/logstore/access_log",
        "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Wildcard authorization

In this example, the source projects are log-project-dev-a, log-project-dev-b, and log-project-dev-c, and the source Logstores are app_a_log, app_b_log, and app_c_log. Replace these with your actual values.

{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:ListShards",
        "log:GetCursorOrData",
        "log:GetConsumerGroupCheckPoint",
        "log:UpdateConsumerGroup",
        "log:ConsumerGroupHeartBeat",
        "log:ConsumerGroupUpdateCheckPoint",
        "log:ListConsumerGroup",
        "log:CreateConsumerGroup"
      ],
      "Resource": [
        "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
    "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
      ],
      "Effect": "Allow"
    }
  ]
}

Attach this policy to the custom RAM role A that you created in Step 1.

Step 3: Grant permissions to write data to the destination Logstore

In Account B (destination Logstore owner), create a custom policy. For example, name it log-etl-target-writer-policy. This policy grants write access to the destination Logstore.

On the Script Editor tab, replace the existing content with a script for exact authorization or wildcard authorization.
Exact authorization

In this example, the destination project is log-project-prod and the destination Logstore is access_log_output. Replace these with your actual values.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*"
      ],
      "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Wildcard authorization
Example: Replace the sample values with your actual values.
- The destination projects are named log-project-dev-a, log-project-dev-b, log-project-dev-c, and so on.
- The destination Logstores are named app_a_log_output, app_b_log_output, app_c_log_output, and so on.
```
{
  "Version": "1",
  "Statement": [
    {
      "Action": [
        "log:Post*",
        "log:BatchPost*"
      ],
       "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
      "Effect": "Allow"
    }
  ]
}
```
Attach this policy to the custom RAM role B that you created in Step 1.

Step 4: Configure the trust policy

In Account B (destination Logstore owner):

Log on to the RAM console.
In the navigation pane on the left, choose Identity Management > Roles.
In the RAM role list, click the destination RAM role.

On the Trust Policy Management tab, click Edit Trust Policy:

{
  "Version": "1",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ID-of-account-A-that-owns-the-source-Logstore@log.aliyuncs.com"
        ]
      }
    }
  ]
}

Note:

Replace "ID-of-account-A-that-owns-the-source-Logstore@log.aliyuncs.com" with the ID of Account A, such as 1234567890123***@log.aliyuncs.com.
This policy allows Account A to obtain temporary credentials from SLS to assume this role in Account B, granting access to resources such as the destination Logstore.
Record the Alibaba Cloud Resource Name (ARN) of this role. You need this ARN when you create the data transformation job in Account A.