Simple Log Service:Process and save data from one logstore to another

Procedure

1. Log on to the Simple Log Service console using an Alibaba Cloud account or a Resource Access Management (RAM) user that has the permissions to create Scheduled SQL jobs.

2. In the Projects section, click the project that contains the source logstore.

3. In the navigation pane on the left, click Log Storage. In the Logstores list, click the name of the logstore that you want to analyze. This logstore is the source logstore for the Scheduled SQL job.

4. Enter a query statement. Then, click Last 15 Minutes to set the time range for the query.

   Note

   This step lets you preview the data for the Scheduled SQL job. Verify that your query statement is correct and that the results contain data.

5. On the Graph tab, click Save as Scheduled SQL Job.

6. Create the Scheduled SQL job.

   1. In the Compute Settings wizard, configure the following parameters and click Next.

      Parameter	Description
      Job Name	The unique name of the Scheduled SQL job. Keep the default name.
      Write Mode	Select Import Data from Logstore to Logstore. This means the job processes data from the source logstore and stores the results in the destination logstore.
      SQL Code	Displays the query statement that you entered in Step 4. You can also define a different query statement here. After you enter the statement, select a time range and click Preview to confirm the results. When the Scheduled SQL job runs, Simple Log Service executes this query statement to analyze the data.
      Target Source and destination logstores in the same account Parameter Description Destination Region Select the region where the destination project is located. Destination Project Select the destination project from the drop-down list. Target Store Select the destination logstore from the drop-down list. Write Authorization The Scheduled SQL job can assume a Default Role or a Custom Role to write data to the destination logstore. Select one of the roles. Default Role: The role has the permissions to run SQL analysis in the source logstore or MetricStore and write the results to the destination logstore or MetricStore. For more information, see Use the default role to create a Scheduled SQL job. Custom Role: Create a custom role and a custom policy for fine-grained permission management. For more information, see Grant the RAM role the permissions to write data to a destination logstore. Source and destination logstores in different accounts Parameter Description Destination Region Select Other Regions and accept the Compliance Warranty On Cross-border Data Transfer. Destination Project Enter the name of the destination project, for example, test-project. Target Store Enter the name of the destination logstore, for example, test-logstore. Write Authorization Select Custom Role. For more information, see Grant the RAM role the permissions to write data to a destination logstore.
      SQL Execution Authorization	The Scheduled SQL job can assume a Default Role or a Custom Role to query and analyze data in the source logstore. Select one of the roles. Default Role: The role has the permissions to run SQL analysis in the source logstore or MetricStore and write the results to the destination logstore or MetricStore. For more information, see Use the default role to create a Scheduled SQL job. Custom Role: Create a custom role and a custom policy for fine-grained permission management. For more information, see Grant a custom RAM role the permissions to analyze logs in a source logstore.

   2. In the Scheduling Settings wizard, configure the following parameters and click OK.

      Parameter	Description
      Scheduling Interval	The frequency at which the job is scheduled. An instance is created for each run, and the interval determines the scheduled time of each instance. Fixed Interval: Schedules the job at a fixed interval. Cron: Schedules the job at an interval specified by a cron expression. A cron expression is accurate to the minute and uses a 24-hour clock. For example, 0 0/1 * * * runs the job every hour, starting at 00:00. To configure a time zone, you must select the Cron mode. For a list of common time zones, see Time zones.
      Scheduling Time Range	The time range during which the job is scheduled. From Specific Time: Specifies the start time for the first job instance. Specific Time Range: Specifies a start and end time. The job runs only within this time range. Note Job instances are scheduled only within this time range. New instances are not created outside this range. The scheduling time range refers to __time__. For more information, see Reserved fields.
      SQL Time Window	The time window for the logs to be analyzed when the job runs. This time window works in conjunction with the scheduling interval. The time window cannot be more than five times the Scheduling Interval and cannot exceed one day. For more information, see Time expression syntax. For example, if Scheduling Interval is Fixed Interval 10 Minutes, Start At is 2021-04-01 00:00:00, Delay Task is 30 Seconds, and SQL Time Window is [@m-10m,@m), the first instance is created at 00:00:30. It analyzes logs from the [23:50:00~00:00:00) time range. For more information, see Scheduling and execution scenarios. Note The SQL time window refers to __time__. For more information, see Reserved fields. If you do not define __time__ in the SQL code, the __time__ value for the log written to the destination logstore defaults to the start time of the scheduling window.
      SQL Timeout	The threshold for automatic retries when an SQL analysis operation fails. If the retry duration exceeds the maximum time or the number of retries exceeds the maximum count, the instance stops and its status changes to FAILED. Manually retry a failed instance. For more information, see Retry a Scheduled SQL job instance.
      Delay Task	The amount of time to delay the execution after the scheduled time. The valid values are 0 to 120 seconds. If there is a delay when writing data to the logstore, use this parameter to ensure data integrity.

7. After you create the Scheduled SQL job, view it by choosing Job Management > Scheduled SQL. For more information, see Manage Scheduled SQL jobs.

SDK example

Create a Scheduled SQL job