Simple Log Service:Distribute logs based on rules

Ensure that the source Logstore has ingested raw logs containing the key fields required for filtering and classification. The following is a sample raw log:

{
  "status": 503,
  "method": "POST",
  "uri": "/api/submit",
  "error_message": "service unavailable",
  "stack_trace": "..."
}

Write SPL rules

Use an SPL rule to process raw logs into multiple result sets. Each result set can be distributed to a different destination.

1. Log on to the Simple Log Service console.

2. Click the name of the source project.

3. In the left-side navigation pane, click Job Management.

4. On the Data Transformation tab, click Create Data Transformation Job, select the source logstore, and then click Confirm.

5. Select a time range for the data and specify the SPL rule.

-- Successful requests (2xx)
.let success_logs = *
  | where cast(status as int) >= 200 and cast(status as int) < 300
  | project __time__, method, uri, status, response_time;
-- Server-side errors (5xx)
.let server_error_logs = *
  | where cast(status as int) >= 500
  | project __time__, method, uri, status, error_message, stack_trace;
-- Output all datasets
$success_logs; $server_error_logs;

Debug the SPL rule and configure the storage destinations:

1. In the Raw Logs section, select log data and click Add Test Data.

2. On the Test Data tab, click Debug SPL to view the Transformation Results.

3. After you confirm that the transformation result is correct, click Save Data Transformation (New).

4. In the Create Data Transformation Job (New) panel, configure the storage destinations as follows:

   • Destination Region: Select the region where the destination project is located.

   • Ensure that the destination project and logstore exist and that you have the required access permissions.

   Configure two storage targets. For the first target, set the name to success, the target project to sls-project-beijing, and the target logstore to access-success. Use the Default Role for authorization and select the success_logs result set. For the second target, set the name to error, the target project to sls-project-beijing, and the target logstore to server-errors. Use the Default Role for authorization and select the server_error_logs result set.

5. Verify the distribution result. After the job starts, query the data in each destination logstore to confirm that the logs are distributed as configured. The following table provides an example.

   Input characteristic	Destination
   status=200	project: sls-project-beijing, logstore: access-success
   status=502	project: sls-project-beijing, logstore: server-errors

One-to-many: Back up error logs

This example writes all error logs to destination logstores across different regions and accounts.

-- All error logs
.let all_errors = *
  | where cast(status as int) >= 400
  | project __time__, method, uri, status, error_message;
-- Output the dataset
$all_errors;

Debug the SPL rule and configure the storage destinations:

1. In the Raw Logs section, select log data and click Add Test Data.

2. On the Test Data tab, click Debug SPL to view the Transformation Results.

3. After you confirm that the transformation result is correct, click Save Data Transformation (New).

4. In the Create Data Transformation Job (New) panel, configure the storage destinations as follows:

   • Destination Region: Select the region where the destination project is located.

   • Role ARN:

     • Operations within the same account: Select the Default Role, which uses the Alibaba Cloud system role AliyunLogETLRole. This role already includes the general permissions to access Simple Log Service resources in the current account and is suitable for scenarios where both the source and destination logstores are in the current account.

     • Cross-account operations: Select Custom Role. If the destination logstore belongs to another Alibaba Cloud account, you must manually create a RAM role, grant it read permissions for the source logstore and write permissions for the destination logstores, and configure a trust policy for the role in the destination account. For more information about authorization, see Configure permissions for data transformation jobs.

   This example configures two storage targets. For the first target, named errorlogs, select China (Beijing) as the target region, set the target project to sls-project-beijing, and set the target logstore to error-logs. Use the Default Role for authorization and select the all_errors result set. For the second target, named backup, select China (Hangzhou) as the target region, set the target project to sls-project-hangzhou, and set the target logstore to error-logs-backup. Use the Custom Role for authorization and select the all_errors result set.

5. Verify the distribution result. After the job starts, query the data in each destination logstore to confirm that the logs are distributed as configured. The following table provides an example.

   Input characteristic	Destination
   status=502	project: sls-project-beijing, logstore: error-logs
   	project: sls-project-hangzhou, logstore: error-logs-backup

Many-to-one: Centralized auditing

This example writes logs from different behaviors to a single destination logstore.

-- Sensitive operation logs
.let audit_logs = *
  | where (uri like '/api/*/delete')  or ( uri like '/api/*/create');
-- Login behavior logs
.let login_logs = *
  | where  uri = '/login' or uri = '/logout';
-- Output the datasets
$audit_logs;$login_logs;

Debug the SPL rule and configure the storage destination:

1. In the Raw Logs section, select log data and click Add Test Data.

2. On the Test Data tab, click Debug SPL to view the Transformation Results.

3. After you confirm that the transformation result is correct, click Save Data Transformation (New).

4. In the Create Data Transformation Job (New) panel, configure the storage destination:

   Set Target Name to audit, select China (Beijing) for Target Region, and enter sls-project-beijing for Target Project and security-audit for Target Logstore. For the Authorization Method, select Default Role. For Result Sets to Write, select both login_logs and audit_logs. Then, click Add.

5. Verify the distribution result. After the job starts, query the data in the destination logstore to confirm that the logs are distributed as configured. The following table provides an example.

   Input characteristic	Destination
   uri = /login	project: sls-project-beijing, logstore: security-audit
   uri=/api/user/delete