Configure log collection to ApsaraMQ for Kafka - Serverless App Engine

Serverless App Engine (SAE) allows you to collect standard output (stdout) of application instances and logs from specified directories to ApsaraMQ for Kafka. Data from ApsaraMQ for Kafka can be migrated to other databases, such as Elasticsearch databases. This topic describes the scenarios, procedures, and formats for collecting logs to ApsaraMQ for Kafka in the SAE console.

Before you start

For SAE
Ensure that each application instance reserves at least 0.25 Core CPU and 250 MB of available resources.
For Kafka
- ApsaraMQ for Kafka is activated and a topic is created. SAE supports instances of ApsaraMQ for Kafka 2.X or later.
- If your ApsaraMQ for Kafka instance is created in a virtual private cloud (VPC) and cannot access the Internet, it must reside in the same VPC of the SAE instance.
- An IP address whitelist of the ApsaraMQ for Kafka instance is configured:
  - If the ApsaraMQ for Kafka instance and the SAE instance reside in the same VPC but are associated with different vSwitches, add the vSwitch CIDR block of the SAE instance to the IP address whitelist of the ApsaraMQ for Kafka instance.
  - To enable all the access within the same VPC of Kafka instance, add 0.0.0.0/0 to the whitelist.

Background

Collecting logs from microservice applications to ApsaraMQ for Kafka enhances the log collection feature of Simple Log Service. This feature is applicable to scenarios where you cannot collect logs to Log Service projects or RAM users cannot view logs in Log Service.

See Limits for more details of the usage limits of ApsaraMQ for Kafka.
See Billing overview for details of ApsaraMQ for Kafka billing.

Configure file log collection

Configure log collection when you create an application

Log on to the SAE console. In the left-side navigation pane, choose Applications > Applications. On the page that appears, select a region in the top navigation bar and a namespace from the Namespace drop-down list, and then click Create Application.
In the Basic Information section, configure the application information and click Next: Advanced Settings.

Expand the Log Settings section, turn on Activate Kafka, and then refer the following table for configuration.

Parameter	Description
Kafka Instance	Select the destination instance.
Collection Log Type	Select a log type. Valid values: File Logs (Log Path In Container): Multiple collection rules can be configured. Container Standard Output Logs: Only one collection rule can be configured. This option is available in the Log Type drop-down list only when your vSwitch resides in one of the recommended zones. For more information, see Change the security group and vSwitch of an application
Log Source	Applied to File Logs only. Enter the directory in which the log file is stored. The directory must contain the name of the log file. Example: /tmp0/cjsc.log. Regular expression is supported. If a directory contains many log files that have the same format, you can specify a log source in the format of /xxx/xxx/xxx/.log. Important* Do not store other important files in the Log Source directory, in case of files in the directory being overwritten.
Kafka Topic Name	Select the existing ApsaraMQ for Kafka topic.

Click Create Application.
Veification.
After you deploy an application, SAE collects and delivers logs to the specified directory based on the configured log collection rule.
In the left-side navigation pane of the application details page, choose Log Management > Persistent Logs. On the Persistent Logs page, click the Log collection to Kafka tab to view collected logs. If logs exist, the log collection rule takes effect.

Configure log collection when you deploy applications

Warning

After you redeploy an application, the application is restarted. To prevent unpredictable errors such as business interruptions, we recommend that you deploy applications during off-peak hours.

The procedure that can be performed to update an application varies based on the number of instances in the application. This section provides an example on how to configure the required features for an application in which the number of instances is greater than or equal to 1. For information about how to update an application in which the number of instances is 0, see Update an application.

Log on to the SAE console. In the left-side navigation pane, choose Applications > Applications. On the page that appears, select a region in the top navigation bar and a namespace from the Namespace drop-down list, and then click the desired application name.
On the Basic Information page of the application, click Deploy Application.

Find and expand the Log Settings section, and turn on Activate Kafka.

Parameter	Description
Kafka Instance	Select the destination instance.
Collection Log Type	Select a log type. Valid values: File Logs (Log Path In Container): Multiple collection rules can be configured. Container Standard Output Logs: Only one collection rule can be configured. This option is available in the Log Type drop-down list only when your vSwitch resides in one of the recommended zones. For more information, see Change the security group and vSwitch of an application
Log Source	Applied to File Logs only. Enter the directory in which the log file is stored. The directory must contain the name of the log file. Example: /tmp0/cjsc.log. Regular expression is supported. If a directory contains many log files that have the same format, you can specify a log source in the format of /xxx/xxx/xxx/.log. Important* Do not store other important files in the Log Source directory, in case of files in the directory being overwritten.
Kafka Topic Name	Select the existing ApsaraMQ for Kafka topic.

After you configure the settings, click OK.
Important
You cannot collect a log file to different ApsaraMQ for Kafka topics at the same time. If you deploy your application in phased release mode or canary release mode and retain the configured log source but change the topic for log collection, your logs are still collected to the original topic until all configurations of your application are deployed.
Veification.
After you deploy an application, SAE collects and delivers logs to the specified directory based on the configured log collection rule.
In the left-side navigation pane of the application details page, choose Log Management > Persistent Logs. On the Persistent Logs page, click the Log collection to Kafka tab to view collected logs. If logs exist, the log collection rule takes effect.

Formats

After you enable Activate Kafka, the collected log is in the following format:

{
    "file":"/home/admin/apache-tomcat-8.5.42/logs/localhost.2022-03-01.log",
    "host":"test-kafka-9527eec8-b2c1-4f03-9178-5dac0fe16d07-*****",
    "message":"01-Mar-2022 15:09:36.016 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath",
    "topic":"test2"
}

Parameters reference:

file indicates the path of the collected log file.
host indicates the name of the instance on which you collect logs.
message indicates the content of the collected logs.
topic indicates the ApsaraMQ for Kafka topic to which the collected logs are delivered.

Collect multi-line logs

In Java applications, an error is reported if logs are automatically merged into one line. The following example shows a Java exception log:

java.lang.RuntimeException: testLog
    at cn.niutong.controller.TestController.heathc(TestController.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
  at ...

If a line break \n is detected, the log data is re-written into a new line. We recommend that you package logs into a JSON string in your program and export the string as a single line.

If you have additional requirements, such as merging multiple lines of logs, join the DingTalk group 32874633 for technical support.

FAQ

Are wildcard characters supported for Log Collection to Kafka?
Yes, you can use asterisks (*) as wildcard characters to match all files in a specific folder. Example: /tmp/logs/*.log.
Why do logs fail to be collected?
Logs may fail to be collected due to network exceptions. To troubleshoot the issue, perform the following steps:
1. Log on to the SAE Webshell. Run the telnet command to query the address of the ApsaraMQ for Kafka instance as network connectivity check. For more information about Webshell, see Use the webshell feature to check the health status of applications.
2. Check the network status.
  - Check whether the SAE application and the ApsaraMQ for Kafka instance reside in the same VPC and whether the IP address whitelist is configured.
  - If the network works fine, join the DingTalk group 32874633 for technical support.