All Products
Search
Document Center

Simple Log Service:Create a data transformation job

Last Updated:Aug 11, 2023

Simple Log Service allows you to create a data transformation job to read data from a source Logstore and write transformed data to one or more destination Logstores. You can also query and analyze the transformed data to create more value. This topic describes how to create a data transformation job in the Simple Log Service console.

Prerequisites

Procedure

  1. Log on to the Log Service console.
  2. Go to the data transformation page.

    1. In the Projects section, click the project that you want to manage.
    2. On the Log Storage > Logstores tab, click the Logstore that you want to manage.
    3. On the query and analysis page, click Data Transformation.

  3. In the upper-right corner of the page, select a time range for the required log data.

    Make sure that the Raw Logs tab displays log data.

  4. In the editor, enter transformation statements.

    For more information about the statement syntax, see Language introduction.

  5. Preview transformation results.

    1. Select Quick.

      You can select either Quick or Advanced. For more information, see Preview mode overview.

    2. Click Preview Data.

      View the results.

      • If data fails to be transformed because the syntax of the transformation statements or configured permissions are invalid, troubleshoot the failure as prompted.

      • If the transformed data is returned as expected, go to 6.

  6. Create a data transformation job.

    1. Click Save as Transformation Rule.

    2. In the Create Data Transformation Rule panel, configure the following parameters and click OK.

      Parameter

      Description

      Rule Name

      The name of the transformation rule.

      Authorization Method

      The method used to authorize the data transformation job to read data from the source Logstore. Valid values:

      • Default Role: authorizes the data transformation job to assume the system role AliyunLogETLRole to read data from the source Logstore.

        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Access data by using a default role.

        Note
        • If the authorization is complete within your Alibaba Cloud account, you can skip this operation.

        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.

      • Custom Role: authorizes the data transformation job to assume a custom role to read data from the source Logstore.

        You must grant the custom role the permissions to read from the source Logstore. Then, you must enter the Alibaba Cloud Resource Name (ARN) of the custom role in the Role ARN field. For more information about authorization, see Access data by using a custom role.

      • AccessKey Pair: authorizes the data transformation job to use the AccessKey pair of an Alibaba Cloud account or a RAM user to read data from the source Logstore.

        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to read from the source Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.

        • RAM user: You must grant the RAM user the permissions to read from the source Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Access data by using AccessKey pairs.

      Storage Target

      Target Name

      The name of the storage destination. Storage Target includes Target Project and Target Logstore.

      You can create multiple storage destinations to store the transformed data in different destination Logstores.

      • You can also use the name parameter of the e_output or e_coutput function in the transformation statements to specify the name of the storage destination. For more information, see e_output and e_coutput.

      • If you do not include the e_output function in the transformation statements, the job writes the transformed data to the Logstore in the storage destination that is numbered 1 by default.

        If you want to configure only one destination Logstore, you do not need to include the e_output function in the transformation statements.

      • If you include the e_output or e_coutput function and set the name, project, and logstore parameters for the function, the job runs based on the parameter settings in the functions even if you configure the Target Project and Target Logstore parameters in this step.

      Target Region

      The region of the project to which the destination Logstore belongs.

      If you want to perform data transformation across regions, we recommend that you use HTTPS for data transmission. This ensures the privacy of log data.

      For cross-region data transformation, the data is transmitted over the Internet. If the Internet connections are unstable, data transformation latency may occur. You can select DCDN Acceleration to accelerate the cross-region data transmission. Before you can select DCDN Acceleration, make sure that the global acceleration feature is enabled for the project. For more information, see Enable the global acceleration feature.

      Note

      You are charged for the amount of Internet traffic that is generated when data after compression is transmitted across regions. For more information, see Billable items of pay-by-feature.

      Target Project

      The name of the project to which the destination Logstore belongs.

      Target Logstore

      The name of the destination Logstore.

      Authorization Method

      The method used to authorize the data transformation job to write transformed data to the destination Logstore. Valid values:

      • Default Role: authorizes the data transformation job to assume the system role AliyunLogETLRole to write transformed data to the destination Logstore.

        You must click You must authorize the system role AliyunLogETLRole. Then, you must configure other parameters as prompted to complete the authorization. For more information, see Access data by using a default role.

        Note
        • If you use a RAM user, you must use an Alibaba Cloud account to assign the AliyunLogETLRole role to the user.

        • If you use an Alibaba Cloud account that has assumed the role, you can skip this operation.

      • Custom Role: authorizes the data transformation job to assume a custom role to write transformed data to the destination Logstore.

        You must grant the custom role the permissions to write to the destination Logstore. Then, you must enter the ARN of the custom role in the Role ARN field. For more information about authorization, see Access data by using a custom role.

      • AccessKey Pair: authorizes the data transformation job to use the AccessKey pair of an Alibaba Cloud account or a RAM user to write transformed data to the destination Logstore.

        • Alibaba Cloud account: The AccessKey pair of an Alibaba Cloud account has permissions to write to the destination Logstore. You can directly enter the AccessKey ID and AccessKey secret of the Alibaba Cloud account in the AccessKey ID and AccessKey Secret fields. For more information about how to obtain an AccessKey pair, see AccessKey pair.

        • RAM user: You must grant the RAM user the permissions to write to the destination Logstore. Then, you can enter the AccessKey ID and AccessKey secret of the RAM user in the AccessKey ID and AccessKey Secret fields. For more information about authorization, see Access data by using AccessKey pairs.

      Processing Range

      Time Range

      The time range within which the data is transformed. Valid values:

      • All: transforms data in the source Logstore from the first log entry until the job is manually stopped.

      • From Specific Time: transforms data in the source Logstore from the log entry that is received at the specified start time until the job is manually stopped.

      • Within Specific Period: transforms data in the source Logstore from the log entry that is received at the specified start time to the log entry that is received at the specified end time.

      Note

      The value of Time Range is based on the time when logs are received.

      Advanced

      Advanced Parameter Settings

      You may need to specify passwords, such as database passwords, in transformation statements. Simple Log Service allows you to add a key-value pair to save the passwords. You can specify the res_local("key") in the statements to reference the passwords.

      You can click the plus sign (+) to add more key-value pairs. For example, you can add config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj, which indicates the ID of the virtual private cloud (VPC) to which an ApsaraDB RDS instance belongs. Advanced Parameter Settings

What to do next

After the data transformation job is created, you can perform the following operations:

  • On the Data Transformation Overview page, view the details and status of the job. You can also perform other operations, such as modifying or stopping the job. For more information, see Manage a data transformation job.

  • In a destination Logstore, perform query and analysis operations. For more information, see Query and analyze logs.