Serverless App Engine:Configure log collection to ApsaraMQ for Kafka

Prerequisites

Before you begin, make sure you have the following:

SAE requirements:

• Each application instance has at least 0.25 Core CPU and 250 MB of available resources

ApsaraMQ for Kafka requirements:

• ApsaraMQ for Kafka is activated and a topic is created. SAE supports ApsaraMQ for Kafka 2.X or later.

• If your Kafka instance is deployed in a virtual private cloud (VPC) without Internet access, the Kafka instance must reside in the same VPC as your SAE application.

• An IP address whitelist is configured for your Kafka instance:

  • If the Kafka instance and the SAE application are in the same VPC but on different vSwitches, add the vSwitch CIDR block of the SAE application to the Kafka whitelist.

  • To allow all traffic within the VPC, add 0.0.0.0/0 to the whitelist.

For Kafka usage limits and billing details, see Limits and Billing overview.

Configure log collection when creating an application

1. Log on to the SAE console. In the left-side navigation pane, choose Applications > Applications. Select a region and a namespace from the Namespace drop-down list, then click Create Application.

2. In the Basic Information section, configure the application information and click Next: Advanced Settings.

3. Expand the Log Settings section and turn on Activate Kafka. Configure the following parameters:

   Parameter	Required	Description
   Kafka Instance	Yes	The destination ApsaraMQ for Kafka instance.
   Collection Log Type	Yes	The type of logs to collect. File Logs (Log Path In Container) supports multiple collection rules. Container Standard Output Logs supports only one collection rule and is available only when your vSwitch is in a recommended zone.
   Log Source	Yes (file logs only)	The full path to the log file inside the container, for example /tmp0/cjsc.log. Regular expressions are supported. To match multiple files with the same format, use a wildcard path such as /xxx/xxx/xxx/*.log. Important Do not store other important files in the log source directory — files in that directory may be overwritten.
   Kafka Topic Name	Yes	The existing ApsaraMQ for Kafka topic to deliver logs to.

4. Click Create Application.

5. Verify the configuration. After the application is deployed, SAE starts collecting logs based on your rules. In the left-side navigation pane of the application details page, choose Log Management > Persistent Logs. Click the Log collection to Kafka tab. If logs appear, the collection rule is active.

Configure log collection when deploying an existing application

This section applies to applications with one or more instances. For applications with zero instances, see Update an application.

1. Log on to the SAE console. In the left-side navigation pane, choose Applications > Applications. Select a region and a namespace, then click the application name.

2. On the Basic Information page, click Deploy Application.

3. In the Log Settings section, turn on Activate Kafka. Configure the parameters as described in the table in the previous section.

4. Click OK.

   Important

   A log file can only be delivered to one Kafka topic at a time. If you use phased release or canary release and change the topic while keeping the same log source, logs continue to be delivered to the original topic until all instances are updated.

5. Verify the configuration. After the application is deployed, choose Log Management > Persistent Logs in the left-side navigation pane. Click the Log collection to Kafka tab to confirm that logs are appearing.

Log format

Each message delivered to ApsaraMQ for Kafka is a JSON object with the following fields:

{
    "file":"/home/admin/apache-tomcat-8.5.42/logs/localhost.2022-03-01.log",
    "host":"test-kafka-9527eec8-b2c1-4f03-9178-5dac0fe16d07-*****",
    "message":"01-Mar-2022 15:09:36.016 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath",
    "topic":"test2"
}

Multi-line logs

SAE treats each line break (\n) as a log record boundary. If your application writes multi-line entries — such as Java exception stack traces — each line is delivered as a separate message. Note that in Java applications, an error is reported if logs are automatically merged into one line:

java.lang.RuntimeException: testLog
    at cn.niutong.controller.TestController.heathc(TestController.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
  at ...

To keep multi-line entries together, serialize them as a single-line JSON string in your application before writing to the log file.

For advanced use cases such as server-side log merging, join DingTalk group 32874633 for technical support.

FAQ

Are wildcard characters supported for log source paths?

Yes. Use an asterisk (*) to match all files in a directory. For example, /tmp/logs/*.log matches all .log files in /tmp/logs/.

Why are logs not being collected?

The most common cause is a network connectivity issue between SAE and the Kafka instance. Check the following:

• Network path: Open SAE Webshell for your application and run a telnet command to the Kafka instance endpoint to test connectivity. For details, see Use the Webshell feature to check application health.

• VPC and whitelist: Confirm that the SAE application and the Kafka instance are in the same VPC, and that the Kafka IP address whitelist includes the SAE vSwitch CIDR block.

If the network is reachable but logs are still missing, join DingTalk group 32874633 for technical support.