Simple Log Service:Create a data transformation job

Procedure

1. Log on to the Simple Log Service console.

2. Go to the data transformation page.

   1. In the Projects section, click the project you want.

   2. On the Log Storage > Logstores tab, click the logstore you want.

   3. On the query and analysis page, click Data Transformation.

3. In the upper-right corner, select a time range for the data.

   Make sure that logs appear on the Raw Logs tab after you select the time range.

4. In the editor, enter a data transformation statement.

   For more information about the statement syntax, see Data transformation syntax.

5. Preview the data.

   1. Click Quick.

      Simple Log Service supports Quick and Advanced preview modes. For more information, see Preview and debug data.

   2. Click Preview Data.

      View the preview results.

      • If the data transformation fails due to an incorrect statement or permission errors, follow the instructions on the page to resolve the issue.

      • If the results are correct, proceed to step 6.

6. Create a data transformation job.

   1. Click Save as Transformation Job (Old Version).

   2. In the Create Data Transformation Job panel, configure the following parameters and click OK.

      Parameter	Description
      Job name	The name of the data transformation job.
      Display name	The display name of the data transformation job.
      Job description	The description of the data transformation job.
      Authorization method	The method used to authorize the data transformation job to read data from the source Logstore. Default Role: The job assumes the AliyunLogETLRole system role to read data from the source logstore. Custom Role: The job assumes a custom role to read data from the source logstore. You must first grant the custom role permissions to read data from the source logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
      Storage destination
      Destination name	The storage destination name. A storage destination includes configurations such as the project and Logstore. You can create multiple storage destinations to store transformation results in different Logstores. Note the following: You can use the name parameter in the e_output() or e_coutput() function within your transformation statement to define the Destination name. For more information, see e_output. If the transformation statement does not include the e_output() function, the job writes results to the first configured destination Logstore by default. If you have only one destination Logstore, the e_output() function is typically not needed. If you specify the name, project, and logstore parameters in the e_output() or e_coutput() function, those settings override the Destination project and Destination logstore configurations in the UI.
      Destination region	The region where the destination project is located. Cross-region data transformation uses HTTPS to ensure data privacy. Cross-region data transformation transmits data over the public internet, which may cause job latency due to network instability. Select the DCDN Acceleration checkbox to accelerate cross-region transmission. Before you use this feature, make sure that acceleration is enabled for the corresponding project. For more information, see Accelerate log collection. Note Cross-region data transformation is billed based on outbound internet traffic (after compression). For more information, see Pay-by-feature.
      Destination project	The project where the transformed data is stored.
      Destination logstore	The Logstore where the transformation results are stored. Warning Do not set the destination Logstore to the same as the source Logstore. This misconfiguration causes an infinite data loop and incurs additional storage and traffic costs. You are responsible for any charges resulting from this error.
      Authorization method	The method used to authorize the data transformation job to write data to the destination Logstore. Default Role: The job assumes the AliyunLogETLRole system role to write transformation results to the destination Logstore. Custom Role: The job assumes a custom role to write transformation results to the destination Logstore. You must first grant the custom role permissions to write data to the destination Logstore. Then, enter the ARN of the custom role in the Role ARN field. For more information, see Use a custom role to access data.
      Processing scope
      Time range	The time range for the data transformation job. Note This time range is based on the time when the logs are received by the service. All: Processes all logs from the earliest time point available in the logstore until the job is manually stopped. From Specific Time: Processes logs from a specified start time until the job is manually stopped. Specific Time Range: Processes logs within a specified time range. The job stops automatically at the end of the range.
      Advanced options
      Advanced parameter configuration	For sensitive information such as database connection passwords used in processing statements, you can save the information as key-value pairs in a secret and reference it in the processing statement by using res_local("key"). Click + to add multiple key-value pairs. For example, config.vpc.vpc_id.test1:vpc-uf6mskb0b****n9yj can represent the ID of the VPC where an RDS instance is located.