This topic describes how to create a data transformation task to transform log data.

Prerequisites

Background information

The data transformation feature of Log Service allows you to read log data from the source Logstore and process and filter the data. You can create a transformation rule to transform raw log data that is continuously updated or generated within a specified time period, and write the transformed log data to multiple Logstores. You can also search and analyze the transformed log data.

Procedure

  1. Log on to the Log Service console.
  2. In the Projects section, click the source project.
  3. Enable the data transformation mode.
    You can enable the data transformation mode in one of the following ways:
    • Choose Log Management > Logstores. Click the source Logstore. On the search and analytics page that appears, turn on the Data Transformation switch. The data transformation page appears.
    • Choose Log Management > Logstores. On the Logstores tab, find the source Logstore, and click the > icon of the Logstore. Choose Data Transformation > Data Transformation. Click the plus sign (+) next to Data Transformation. The data transformation page appears.
  4. Create a data transformation rule.
    1. Set the query time range of the raw log data.
    2. Enter a transformation rule in the text box.
      For more information, see Data transformation syntax.
    3. Click Preview Data.
    4. On the Add Preview Settings page, set the parameters, and then click OK. The following table describes the parameters.
      Note When you preview data for the first time, you must set the parameters. After you set the parameters, you can click Modify Preview Settings to modify the parameters.
      Parameter Description
      AccessKey ID The AccessKey ID of the Alibaba Cloud account or RAM user that is granted read permissions on the source Logstore. For more information, see Create an AccessKey pair for a RAM user to access the source Logstore.
      AccessKey Secret The AccessKey secret of the Alibaba Cloud account or RAM user that is granted read permissions on the source Logstore. For more information, see Create an AccessKey pair for a RAM user to access the source Logstore.
      Advanced Parameter Settings Set the passwords that are required in the transformation rule in the key-value pair format. For example, you can set the password that is used to connect to a database in the key-value pair format. Passwords are referenced in the transformation rule syntax by using the ${key} variable.
  5. View transformed data.
    • If the data transformation task fails because the transformation rule syntax or the specified AccessKey pair is invalid, troubleshoot the issue as prompted.
    • If the transformed data is returned as expected, perform the operations in step 6.
  6. Save the data transformation rule.
    1. Click Save as Transformation Rule.
    2. On the Create Data Transformation Rule page, set the parameters, and then click OK. The following table describes the parameters.
      You can store transformation results in multiple destination Logstores.
      Note
      • If you need to store transformed data in multiple destination Logstores, you can use the e_output or e_couput function in the data transformation rule. You can use the name and logstore parameters to write specific data to specific destination Logstores.
      • If you need to store transformed data in multiple destination Logstores, but do not use the e_output function in the transformation rule, the transformed data is written to the first destination Logstore. If you need to store transformed data in only one destination Logstore, you do not need to use the e_output function in the transformation rule.
      Parameter Description
      Rule Name The name of the transformation rule.
      AccessKey The AccessKey ID of the Alibaba Cloud account or RAM user that is granted read permissions on the source Logstore. For more information, see Create an AccessKey pair for a RAM user to access the source Logstore.
      AccessKey Secret The AccessKey secret of the Alibaba Cloud account or RAM user that is granted read permissions on the source Logstore. For more information, see Create an AccessKey pair for a RAM user to access the source Logstore.
      Storage Target The logical name of the storage target.
      Target Project The name of the destination project to which the transformed data is written.
      Target Logstore The name of the destination Logstore to which the transformed data is written.
      AccessKey The AccessKey ID of the Alibaba Cloud account or RAM user that is granted write permissions on the destination Logstore. For more information, see Create an AccessKey pair for a RAM user to access the destination Logstore.
      AccessKey Secret The AccessKey secret of the Alibaba Cloud account or RAM user that is granted write permissions on the destination Logstore. For more information, see Create an AccessKey pair for a RAM user to access the destination Logstore.
      Time Range Valid values:
      • All: transforms data in the source Logstore from the first log entry until the transformation task is manually stopped.
      • From Specific Time: transforms data in the source Logstore from the position that corresponds to the specified start time until the transformation task is manually stopped.
      • Within Specific Period: transforms data in the source Logstore from the position that corresponds to the specified start time to the position that corresponds to the specified end time.
      Note The time range that is specified by the Time Range parameter is subject to the log receiving time.
      Advanced Parameter Settings Set the passwords that are required in the transformation rule in the key-value pair format. For example, you can set the password that is used to connect to a database in the key-value pair format. Passwords are referenced in the transformation rule syntax by using the ${key} variable.

      After you save the transformation rule, the rule is automatically executed.

Result

After you create a data transformation task, you can manage the task on the Data Transformation Overview page. For more information, see Manage a data transformation task.