How to collect multiple copies of logs from a file - Simple Log Service

By default, a file can match only one Logtail configuration. This topic describes how to collect logs from a single file multiple times.

Background

If multiple Logtail configurations match the same file, Logtail consumes multiple times the CPU, memory, disk I/O, and network I/O resources. This can degrade the performance of other services on the server. Therefore, by default, a file can match only one Logtail configuration.

Solutions

Store log data in different Logstores

Instead of creating multiple Logtail configurations, you can use the data transformation feature of Simple Log Service (SLS) to replicate logs. This method avoids affecting the performance of other services on the host. For more information, see Replicate data from a Logstore.

Create multiple Logtail configurations

Host scenarios

Create a symbolic link to the directory

Create a symbolic link to the directory that contains the file. Use the original path in one Logtail configuration and the symbolic link path in another. For example, run the following command to create a symbolic link named /home/log/nginx/link_log for the directory /home/log/nginx/log. You can then access files in the directory through the symbolic link.
```
ln -s /home/log/nginx/log /home/log/nginx/link_log
```
Enable forced collection

In the Input Configurations section of the Logtail configuration, turn on the Allow File to Be Collected for Multiple Times switch. For more information, see Collect text logs from servers and Manually install Logtail to collect text logs from servers.

Container scenarios

To configure container log collection, see Collect text logs from an ACK cluster in DaemonSet mode and Collect container logs (standard output and files) from a cluster using the console.

Console

In the Input Configurations section of the Logtail configuration, turn on the Allow File to Be Collected for Multiple Times switch.
Custom resource definition (CRD)

Define multiple CRDs.