All Products
Search
Document Center

Simple Log Service:Access data by using AccessKey pairs

Last Updated:Aug 11, 2023

For a data transformation job, you can specify the AccessKey pairs of Alibaba Cloud accounts or RAM users to authorize Simple Log Service to read data from a source Logstore and write transformed data to one or more destination Logstores. The AccessKey pair of an Alibaba Cloud account has access permissions on Logstores and can be directly used. If you use a RAM user, you must grant the RAM user the access permissions on Logstores. For more information, see the following sections.

Prerequisites

RAM users are created. For more information, see Create a RAM user.

Important
  • When you create the RAM users, select OpenAPI Access for Access Mode. Then, record the AccessKey pairs of the RAM users.

  • The AccessKey secret of a RAM user is displayed only when you create the AccessKey pair for the RAM user. You cannot query the AccessKey secret after you create the AccessKey pair. Keep your AccessKey secret confidential.

Grant a RAM user the permissions to read from a source Logstore

After you use an Alibaba Cloud account to grant the read permissions to the RAM user, the RAM user can read from the source Logstore. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.

  1. Log on to the RAM console by using your Alibaba Cloud account.

  2. Create a policy.

    The policy allows the RAM user to read from a source Logstore.

    1. In the left-side navigation pane, choose Permissions > Policies.

    2. On the Policies page, click Create Policy.

    3. On the Create Policy page, configure the following information and click OK.

      Configuration item

      Description

      Configuration mode

      Click JSON.

      Policy document

      Replace the content in the editor with the following script.

      • Policy that uses exact match

        In this example, the source project name is log-project-prod, and the source Logstore name is access_log. Replace the project and Logstore names based on your business requirements.

        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:ListShards",
                "log:GetCursorOrData",
                "log:GetConsumerGroupCheckPoint",
                "log:UpdateConsumerGroup",
                "log:ConsumerGroupHeartBeat",
                "log:ConsumerGroupUpdateCheckPoint",
                "log:ListConsumerGroup",
                "log:CreateConsumerGroup"
              ],
              "Resource": [
                "acs:log:*:*:project/log-project-prod/logstore/access_log",
                "acs:log:*:*:project/log-project-prod/logstore/access_log/*"
              ],
              "Effect": "Allow"
            }
          ]
        }
      • Policy that uses fuzzy match

        In this example, the source project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the source Logstore name can be app_a_log, app_b_log, or app_c_log. Replace the project and Logstore names based on your business requirements.

        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:ListShards",
                "log:GetCursorOrData",
                "log:GetConsumerGroupCheckPoint",
                "log:UpdateConsumerGroup",
                "log:ConsumerGroupHeartBeat",
                "log:ConsumerGroupUpdateCheckPoint",
                "log:ListConsumerGroup",
                "log:CreateConsumerGroup"
              ],
              "Resource": [
                "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log",
            "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log/*"
              ],
              "Effect": "Allow"
            }
          ]
        }

        For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.

      Policy name

      Enter a name for the policy. In this example, enter log-etl-source-reader-1-policy.

  3. Attach the policy to the RAM user.

    1. In the left-side navigation pane, choose Identities > Users.

    2. On the Users page, find the RAM user to which you want to attach the policy and click Add Permissions in the Actions column.

    3. In the Add Permissions panel, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-source-reader-1-policy.

      Configure permissions
    4. Confirm the authorization results. Then, click Complete.

Grant a RAM user the permissions to write to destination Logstores

After you use an Alibaba Cloud account to grant the write permissions to the RAM user, the RAM user can write to the destination Logstores. When you create a data transformation job, you can enter the AccessKey pair of the RAM user. For more information, see Create a data transformation job.

  1. Log on to the RAM console by using your Alibaba Cloud account.

  2. Create a policy.

    The policy allows the RAM user to write to the destination Logstores.

    1. In the left-side navigation pane, choose Permissions > Policies.

    2. On the Policies page, click Create Policy.

    3. On the Create Policy page, configure the following information and click OK.

      Configuration item

      Description

      Configuration mode

      Click JSON.

      Policy document

      Replace the content in the editor with the following script.

      • Policy that uses exact match

        In this example, the destination project name is log-project-prod, and the destination Logstore name is access_log_output. Replace the project and Logstore names based on your business requirements.

        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:Post*",
                "log:BatchPost*"
              ],
               "Resource": "acs:log:*:*:project/log-project-prod/logstore/access_log_output",
              "Effect": "Allow"
            }
          ]
        }
      • Policy that uses fuzzy match

        In this example, the destination project name can be log-project-dev-a, log-project-dev-b, or log-project-dev-c, and the destination Logstore name can be app_a_log_output, app_b_log_output, or app_c_log_output. Replace the project and Logstore names based on your business requirements.

        {
          "Version": "1",
          "Statement": [
            {
              "Action": [
                "log:Post*",
                "log:BatchPost*"
              ],
               "Resource": "acs:log:*:*:project/log-project-dev-*/logstore/app_*_log_output",
              "Effect": "Allow"
            }
          ]
        }

        For more information about authorization scenarios, see Use custom policies to grant permissions to a RAM user.

      Policy name

      Enter a name for the policy. In this example, enter log-etl-target-writer-1-policy.

  3. Attach the policy to the RAM user.

    1. In the left-side navigation pane, choose Identities > Users.

    2. On the Users page, find the RAM user to which you want to attach the policy and click Add Permissions in the Actions column.

    3. In the Add Permissions panel, click the Custom Policy tab. From the list of custom policies, click the policy that you created in Step 2 and click OK. In this example, the policy is log-etl-target-writer-1-policy.

      Write permissions
    4. Confirm the authorization results. Then, click Complete.

What to do next

You can specify the AccessKey pairs of the RAM users for a data transformation job. For more information, see Create a data transformation job.

Modify a transformation rule