Collect SAE file logs to SLS for management and analysis - Serverless App Engine

You can collect standard output logs or file logs from SAE applications and send them to Simple Log Service (SLS) for persistent storage and aggregated analysis.

Prerequisites

Activate Simple Log Service. If you have not activated the service, log on to the Simple Log Service console and follow the on-screen instructions to activate it.
Ensure that each instance in the application has at least 0.25 Core CPU and 250 MB of available memory.

Enable log collection

Procedure

Create an application
On the SAE Application List page, select a destination region and namespace in the top navigation bar, and then click Create Application. On the Application Basic Information page, configure the parameters and click Next: Advanced Settings.
Change a running application
Warning
After you redeploy an application, the application is restarted. To prevent unpredictable errors such as business interruptions, we recommend that you deploy applications during off-peak hours.
On the SAE Application List page, select a destination region and namespace in the top navigation bar. Click the ID of the target Application to open the application details page.In the navigation pane on the left, click Basic Information. In the upper-right corner, click Deploy Application.
Change a stopped application
On the SAE Application List page, select a destination region and namespace in the top navigation bar. Click the ID of the target Application to open the application details page.Click Basic Information, and then click Modify Application Configuration.
Expand the Log Settings section and enable Activate Simple Log Service.

Configuration description

Select an SLS resource to store the logs of your SAE application:
- Create Simple Log Service Resource: The system automatically creates an SLS project. The project name uses the format sae-<random_string>.
- Use Existing Simple Log Service Resources: Select an existing SLS project. The project must be in the same region as the application.
Log label: Log label are used to identify and categorize log entries. After you enable Enable custom log labels, select one of the following Type to configure the key and value. To configure multiple key-value pairs, click Add.
- Custom: Enter a Variable Name and a Variable Value.
- Reference Configuration Item: Enter a Variable Name, and then select an existing Configuration Item Name and Key.
Important
This feature is in invitational preview. To use this feature, contact the relevant technical staff in the DingTalk group (ID: 32874633) to add your account to the whitelist.
Collection Log Type:
- File Logs (Log Path in Container): This option applies to scenarios where the application writes logs to a file. In the Log Source field, specify the absolute path of the log file in the container, including the file name. For example, /tmp/test.log. Wildcard matching is supported. For example, /path/to/your/files/*.log.
- Container Standard Output Logs: This option applies to scenarios where the application prints logs to standard output. The Log Source is displayed as stdout.log.
- Click + Add to add multiple records.
- If you select Use Existing Simple Log Service Resources, you can configure a logstoreName and a logtail for each record.
Important
- To prevent files in the directory from being overwritten, do not store other important files in the Log Source path. For example, if the application's executable file and data are stored in /home/admin/app/, setting the log storage path to /home/admin/ overwrites all directories and files under that path. To prevent this, set the log storage path to /home/admin/log/.
- During a batch or phased release, if you only change the project or Logstore, logs will still be collected to the old destination until all batches are deployed. This is because SLS does not support collecting logs from the same file to different projects or Logstores.

Verify the results

In the SAE Application List, select the destination region and namespace from the top navigation bar. Click the target Application ID to go to the application details page.
In the navigation pane on the left, choose Log Management > Persistent Logs > Log Collection to Simple Log Service to view the SLS logs.
In the Actions column, click File Logs to go to the corresponding Logstore and view the log details.

Remove prefixes from log content

When you collect container standard output logs, the original log content includes an unnecessary prefix. You can use one of the following methods to remove the prefix.

Method 1: Customize the Logtail configuration

Important

This feature is available only for Logtail 2.0 and later. To use this feature, contact a product technical expert in the DingTalk group (ID: 32874633) to upgrade the Logtail version for your SAE application.

Expand the corresponding Logstore and click Logtail Configurations.
On the Logtail Configuration page, click the target Logtail configuration item.
Click Edit and configure the following parameters. Keep the default values for other parameters.
1. Expand the Processor Configurations section. Set Processing Method If Splitting Fails to Retain Single Line.
2. Set Processing Method to SPL.
3. In the SPL Statement text box, enter * | extend msg=regexp_replace(content, '(\S{33,36}\s\S+\sF\s)','') | project-away content.
Click Save.
The prefix is removed from newly collected logs.

Method 2: Redirect standard output

Create an application
On the SAE Application List page, select a destination region and namespace in the top navigation bar, and then click Create Application. On the Application Basic Information page, configure the parameters and click Next: Advanced Settings.
Change a running application
Warning
After you redeploy an application, the application is restarted. To prevent unpredictable errors such as business interruptions, we recommend that you deploy applications during off-peak hours.
On the SAE Application List page, select a destination region and namespace in the top navigation bar. Click the ID of the target Application to open the application details page.In the navigation pane on the left, click Basic Information. In the upper-right corner, click Deploy Application.
Change a stopped application
On the SAE Application List page, select a destination region and namespace in the top navigation bar. Click the ID of the target Application to open the application details page.Click Basic Information, and then click Modify Application Configuration.
Expand the Startup Command section and configure the startup command to redirect the standard output:
For an application that is deployed from an image, add 2>&1 | tee -a /home/admin/log/std.log after the original startup command. In this command, /home/admin/log/std.log is the log file storage path, which you can customize.
For example, if the original startup command in the image is CMD ["./main"], configure the startup command as ./main 2>&1 | tee -a /home/admin/log/std.log.
If the image environment does not support the tee command, or if you do not need to view real-time logs in the console, replace 2>&1 | tee -a /home/admin/log/std.log with > /home/admin/log/std.log 2>&1.
Expand the Log Configuration section. Set Log Collection Type to File Log (Log Path In Container). Set Log Source to the log file storage path, such as /home/admin/log/std.log.
Click OK and wait for the application to be deployed.
The prefix is removed from newly collected logs.

Set environment variables to improve Logtail collection performance

You can configure Logtail startup parameters by setting environment variables in the SAE console. This improves Logtail collection performance and helps resolve bottlenecks that affect log collection, such as high memory usage, high log data traffic, and high data sending rates.

In the Environment Variable section, select Custom or Reference Configuration Item. Then, enter the Variable Name and Variable Value/Variable Reference. For more information, see Set environment variables.

Variable name rules
Add the sls_ prefix to the Logtail environment variable that you want to configure. For example, as shown in the following figure, to add the Logtail parameter max_read_buffer_size, you must configure the environment variable as sls_max_read_buffer_size. The figure shows that the maximum size for reading each log is 524,288 bytes. For more information about the parameters, see Set Logtail startup parameters.
Mapping between environment variables and Logtail startup parameters
For more information, see Environment variable descriptions.

Quotas and limits

An Alibaba Cloud account can have a maximum of 200 Logstore resources and 50 project resources. The limits on basic Simple Log Service resources can affect log collection for SAE applications. For example, these limits can lead to short log retention periods or failed log collection. For more information about the limits, see Basic resources.

Billing description

Simple Log Service is a paid service. For more information, see SLS billing description.

FAQ

Does the SAE log feature support log rotation?
Yes, it does. You can rotate logs by adding a wildcard character to the log file name.
The supported wildcard characters are the asterisk (*) and the question mark (?).
When I view file logs, no data is displayed in the SLS console. What should I do?
By default, SLS queries data from the last 15 minutes. If no data is displayed in the SLS console, use Webshell to check whether your application has generated file log information. For more information, see Use Webshell to diagnose applications.
- If log information exists, join the DingTalk group (ID: 32874633) and contact a product technical expert for assistance.
- If no log information exists, check your application configuration.