Realtime Compute for Apache Flink:Best practice of secure access

Solution overview

This solution implements more secure access by using an AccessKey pair based on the principle of least privilege. This solution uses the AccessKey pair of a RAM user that is granted the permissions to access specific upstream and downstream resources or the minimum required permissions, instead of the AccessKey pair of an Alibaba Cloud account. In addition, the AccessKey pair is encrypted by using the variables of Flink to reduce the risk of AccessKey pair leaks in plaintext. This reduces the security risks of all cloud resources within your Alibaba Cloud account.

In this topic, a RAM user is created. The RAM user is granted the permissions on a specific Logstore in a project of Simple Log Service. The AccessKey pair of the RAM user is encrypted by using the variables of Flink and used to read data from and write data to Simple Log Service.

Procedure

1. Create a RAM user by using your Alibaba Cloud account or as a RAM administrator. For more information, see Create a RAM user.

   You must select OpenAPI Access in the Access Mode section. After that, an AccessKey ID and an AccessKey secret are automatically generated for the RAM user.

   Important

   The AccessKey secret of a RAM user is displayed only when the RAM user is created. Keep the AccessKey secret confidential.

2. Grant permissions on Simple Log Service to the RAM user.

   1. Create custom policies.

      1. In the left-side navigation pane, choose Permissions > Policies.

      2. On the Policies page, click Create Policy.

      3. On the Create Policy page, click the JSON tab, replace the existing script in the code editor with the following sample code, and then click Next to edit policy information.

         Replace the project and logstore parameters in the sample code based on your business requirements. For more information about how to create a custom policy, see Examples of using custom policies to grant permissions to a RAM user.

         Grant the read-only permissions on a specific Logstore

         {
             "Version": "1",
             "Statement": [
                 {
                     "Action": "log:ListProject",
                     "Resource": "acs:log:*:*:project/*",
                     "Effect": "Allow"
                 },
                 {
                     "Action": "log:List*",
                     "Resource": "acs:log:*:*:project/<Project name>/logstore/*",
                     "Effect": "Allow"
                 },
                 {
                     "Action": [
                         "log:Get*",
                         "log:List*"
                     ],
                     "Resource": "acs:log:*:*:project/<Project name>/logstore/<Logstore name>",
                     "Effect": "Allow"
                 }
             ]
         }

         Grant the write permissions on a specific Logstore

         To grant a RAM user only the permissions to write data to a specified Logstore, use the following policy.

         When you create a policy, you can specify either a Logstore or a Metricstore after the logstore keyword. If you want to manage Metricstores, the following policy also applies.

         {
           "Version":"1",
           "Statement":[
             {
               "Effect":"Allow",
               "Action":[
                 "log:PostLogStoreLogs"
               ],
               "Resource":[
                 "acs:log:*:*:project/<Project name>/logstore/<Logstore name>"
               ]
             }
           ]
         }

      4. Enter the name and description for the policy and click OK.

3. Use the custom policies that are created in the previous step to grant permissions to the RAM user. For more information, see Grant permissions to a RAM user.

4. Configure variables to reduce the risk of AccessKey pair leaks in plaintext.

   Create variables for the AccessKey ID and AccessKey secret that you obtain when you create the RAM user in Step 1. After the variables are created, you need to only call the variable names instead of specifying the AccessKey ID and AccessKey secret. For more information about how to create variables, see Manage variables and keys. In this topic, variables named slslak and slsaks are created for the AccessKey ID and AccessKey secret.

5. Create a Flink deployment to read data from Simple Log Service.

   You can call the variables in the ${secret_values.Variable name} format when you develop an SQL draft. This prevents security risks caused by AccessKey pair leaks in plaintext. Sample code:

   CREATE TEMPORARY TABLE sls_input(
      `__source__` STRING METADATA VIRTUAL,
      __tag__ MAP<VARCHAR, VARCHAR> METADATA VIRTUAL,
      `__topic__` STRING METADATA VIRTUAL,
      deploymentName STRING,
      `level`STRING,
      `location` STRING,
      message STRING,
      thread STRING,
      `time`STRING
   ) WITH (
     'connector' = 'sls',
     'endpoint' ='cn-beijing-intranet.log.aliyuncs.com',
     'accessId' = '${secret_values.slsak}',
     'accessKey' = '${secret_values.slsaks}',
     'starttime' = '2024-08-30 15:39:00',
     'project' ='test',
     'logstore' ='flinktest'
   );
   
   CREATE TEMPORARY TABLE blackhole_sink(
      `__source__` STRING,
      `__topic__` STRING,
      deploymentName STRING,
      `level` STRING,
      `location` STRING,
      message STRING,
      thread STRING,
      `time` STRING,
      receive_time BIGINT
   ) WITH (
     'connector' = 'blackhole'
   );
   
   INSERT INTO blackhole_sink
   SELECT `__source__`,
      `__topic__`,
      deploymentName,
      `level`,
      `location`,
      message,
      thread,
      `time`,
      cast(__tag__['__receive_time__'] as bigint) as receive_time
   FROM sls_input; 
   

References

• To use Flink to read data from and write data to Object Storage Service (OSS), you can configure custom policies to grant the read and write permissions on a specific bucket to a RAM user. For more information, see RAM policies.

• For more information about how to develop an SQL draft, see Develop an SQL draft.

• Flink provides job and project variables to prevent security risks caused by the leak of sensitive data such as an AccessKey pair and a password in plaintext. You can use these variables in multiple scenarios. For more information about how to create variables, see Manage variables and keys.