Collect logs from a file multiple times

By default, a file matches only one Logtail configuration. Use data transformation or symbolic links to collect the same file into multiple destinations.

Background

When multiple Logtail configurations match the same file, Logtail consumes extra CPU, memory, disk I/O, and network I/O resources, which degrades other services on the server. Therefore, a file matches only one Logtail configuration by default.

Solutions

Store log data in different Logstores

Instead of creating multiple Logtail configurations, use the SLS data transformation feature to replicate data from a Logstore. This avoids extra resource consumption on the host.

Create multiple Logtail configurations

Host scenarios

Create a symbolic link to the directory

Create a symbolic link to the log file directory. Use the original path in one Logtail configuration and the symlink path in the other. For example, create a symlink named /home/log/nginx/link_log for /home/log/nginx/log.
```
ln -s /home/log/nginx/log /home/log/nginx/link_log
```
Enable forced collection

In the Input Configurations section, enable the Allow File to Be Collected for Multiple Times toggle. Collect text logs from servers and Manually install Logtail to collect text logs from servers.

Container scenarios

Container log collection supports two approaches: Collect text logs from an ACK cluster in DaemonSet mode and Collect container logs (standard output and files) from a cluster using the console.

Console

In the Input Configurations section, enable the Allow File to Be Collected for Multiple Times toggle.
Custom Resource Definition (CRD)

Define multiple CRDs.