For a data transformation job, you can specify the AccessKey pairs of Alibaba Cloud accounts or RAM users to authorize Simple Log Service to read data from a source Logstore and write transformed data to one or more destination Logstores. The AccessKey pair of an Alibaba Cloud account has access permissions on Logstores and can be directly used. If you use a RAM user, you must grant the RAM user the access permissions on Logstores. For more information, see the following sections.
Prerequisites
RAM users are created. For more information, see Create a RAM user.
When you create the RAM users, select OpenAPI Access for Access Mode. Then, record the AccessKey pairs of the RAM users.
The AccessKey secret of a RAM user is displayed only when you create the AccessKey pair for the RAM user. You cannot query the AccessKey secret after you create the AccessKey pair. Keep your AccessKey secret confidential.
Grant a RAM user the permissions to read from a source Logstore
After you use an Alibaba Cloud account to grant the read permissions to the RAM user, the RAM user can read from the source Logstore. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.
Log on to the RAM console by using your Alibaba Cloud account.
Create a policy.
The policy allows the RAM user to read from a source Logstore.
In the left-side navigation pane, choose .
On the Policies page, click Create Policy.
On the Create Policy page, configure the following information and click OK.
Configuration item
Description
Configuration mode
Click JSON.
Policy document
Replace the content in the editor with the following script.
Policy that uses exact match
In this example, the source project name is log-project-prod, and the source Logstore name is access_log. Replace the project and Logstore names based on your business requirements.
{ "Version": "1", "Statement": [ { "Action": [ "log:ListShards", "log:GetCursorOrData", "log:GetConsumerGroupCheckPoint", "log:UpdateConsumerGroup", "log:ConsumerGroupHeartBeat", "log:ConsumerGroupUpdateCheckPoint", "log:ListConsumerGroup", "log:CreateConsumerGroup" ], "Resource": [ "acs:log:*:*:project/log-project-prod/logstore/access_log", "acs:log:*:*:project/log-project-prod/logstore/access_log/*" ], "Effect": "Allow" } ] }
Policy that uses fuzzy match
In this example, the source project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the source Logstore name can be app_a_log, app_b_log, or app_c_log. Replace the project and Logstore names based on your business requirements.
{ "Version": "1", "Statement": [ { "Action": [ "log:ListShards", "log:GetCursorOrData", "log:GetConsumerGroupCheckPoint", "log:UpdateConsumerGroup", "log:ConsumerGroupHeartBeat", "log:ConsumerGroupUpdateCheckPoint", "log:ListConsumerGroup", "log:CreateConsumerGroup" ], "Resource": [ "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log", "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*" ], "Effect": "Allow" } ] }
For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.
Policy name
Enter a name for the policy. In this example, enter log-etl-source-reader-1-policy.
Attach the policy to the RAM user.
In the left-side navigation pane, choose .
On the Users page, find the RAM user to which you want to attach the policy and click Add Permissions in the Actions column.
In the Add Permissions panel, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-source-reader-1-policy.
Confirm the authorization results. Then, click Complete.
Grant a RAM user the permissions to write to destination Logstores
After you use an Alibaba Cloud account to grant the write permissions to the RAM user, the RAM user can write to the destination Logstores. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.
Log on to the RAM console by using your Alibaba Cloud account.
Create a policy.
The policy allows the RAM user to write to the destination Logstores.
In the left-side navigation pane, choose .
On the Policies page, click Create Policy.
On the Create Policy page, configure the following information and click OK.
Configuration item
Description
Configuration mode
Click JSON.
Policy document
Replace the content in the editor with the following script.
Policy that uses exact match
In this example, the destination project name is log-project-prod, and the destination Logstore name is access_log_output. Replace the project and Logstore names based on your business requirements.
{ "Version": "1", "Statement": [ { "Action": [ "log:Post*", "log:BatchPost*" ], "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output", "Effect": "Allow" } ] }
Policy that uses fuzzy match
In this example, the destination project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the destination Logstore name can be app_a_log_output, app_b_log_output, or app_c_log_output. Replace the project and Logstore names based on your business requirements.
{ "Version": "1", "Statement": [ { "Action": [ "log:Post*", "log:BatchPost*" ], "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output", "Effect": "Allow" } ] }
For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.
Policy name
Enter a name for the policy. In this example, enter log-etl-target-writer-1-policy.
Attach the policy to the RAM user.
In the left-side navigation pane, choose .
On the Users page, find the RAM user to which you want to attach the policy and click Add Permissions in the Actions column.
In the Add Permissions panel, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-target-writer-1-policy.
Confirm the authorization results. Then, click Complete.
What to do next
You can specify the AccessKey pairs of the RAM users for a data transformation job. For more information, see Create a data transformation job.
In Section 1, enter the AccessKey pair of the RAM user that is granted the permissions to read from a source Logstore. For more information, see Grant a RAM user the permissions to read from a source Logstore.
In Section 2, enter the AccessKey pair of the RAM user that is granted the permissions to write to destination Logstores. For more information, see Grant a RAM user the permissions to write to destination Logstores.