This topic describes the limits of Logtail, including the limits on log files, checkpoints, Logtail configurations, resources, performance metrics, and troubleshooting.

Table 1. Limits on log files
Item Description
Log file encoding Logtail supports log files that are encoded in UTF-8 and GBK. We recommend that you use UTF-8-encoded log files to improve processing performance. If log files are encoded in other formats, errors such as garbled characters and data loss may occur.
Log file size The log file size is unlimited.
Log file rotation Logtail supports log file rotation. Both .log* and .log files are supported in rotation.
Log collection behavior performed when log parsing is blocked When log parsing is blocked, Logtail keeps the log file descriptor (FD) open. If log file rotation occurs multiple times during the blocking period, Logtail attempts to ensure that new log files are parsed in sequence. If the number of new files that are not parsed exceeds 20, Logtail does not process the excess log files.
Symbolic link Monitored directories can be symbolic links.
Log size The maximum size of a log is 512 KB. If a multi-line log is split by using a regular expression to match the first line, the maximum size of each log after splitting is still 512 KB. If the size of a log exceeds 512 KB, the log is forcibly split into multiple parts for collection. For example, if the size of a log is 1,025 KB, the log is split into three parts: 512 KB, 512 KB, and 1 KB. Then, the log parts are collected in sequence.
Regular expression Logtail uses regular expressions that are compatible with Perl.
Multiple Logtail configurations for the same log file You cannot use multiple Logtail configurations for the same log file. We recommend that you store data collected from a log file to the same Logstore. You can configure multiple tasks to subscribe to logs collected from different files. If you want to use multiple Logtail configurations for the same log file, configure symbolic links for log files to bypass this limit.
File opening behavior When Logtail collects data from a log file, Logtail keeps the log file open. If the log file is not modified for more than 5 minutes and log rotation does not occur, Logtail closes the log file.
First log collection behavior Logtail collects data only from incremental log files. If the size of a log file exceeds 1 MB the first time the modification to the log file is detected, Logtail collects data from the last 1 MB. If the log file size does not exceed 1 MB, Logtail collects data from the beginning of the log file. If the log file is not modified after the Logtail configuration is delivered, Logtail does not collect data from the log file.
Non-standard text logs If a log contains \0, the log is truncated at the first occurrence of \0.
Table 2. Limits on checkpoints
Item Description
Checkpoint timeout period If a log file is not modified for more than 30 days, the checkpoint of the log file is deleted.
Checkpoint storage policy Checkpoints are stored at intervals of 15 minutes and at the point in time when Logtail exits. For more information about how to change the values of the related parameters, see Set the startup parameters of Logtail.
Checkpoint storage path By default, checkpoints are stored in the /tmp/logtail_checkpoint directory. For more information about how to change the values of the related parameters, see Set the startup parameters of Logtail.
Table 3. Limits on Logtail configurations
Item Description
Configuration update A configuration update requires approximately 30 seconds to take effect.
Dynamic loading of Logtail configurations Logtail configurations can be dynamically updated. An update of a Logtail configuration does not affect other Logtail configurations.
Number of Logtail configurations The number of Logtail configurations is unlimited. However, we recommend that you create no more than 100 Logtail configurations for a server.
Multi-tenant isolation Logtail configurations are isolated.
Table 4. Limits on resources and performance metrics
Item Description
Throughput for log processing The default transmission speed of raw logs is limited to 20 MB/s. Log data is uploaded after it is encoded and compressed. The compression ratio ranges from 5:1 to 10:1. If the speed exceeds the limit, log data may be lost. For more information about how to change the values of the related parameters, see Set the startup parameters of Logtail.
Maximum processing speed for logs Single-core processing speed: The maximum processing speed is 100 MB/s in simple mode, 40 MB/s in delimiter mode, and 30 MB/s in JSON mode. By default, the maximum processing speed is 20 MB/s in full regex mode. The maximum processing speed in full regex mode varies based on the complexity of regular expressions. If multiple processing threads are started, the performance can be improved by 150% to 300%.
Number of monitored directories Logtail limits the depth of monitored directories to reduce the consumption of user resources. If the upper limit is reached, Logtail stops monitoring additional directories or log files. Logtail can monitor a maximum of 3,000 directories, including subdirectories.
Number of monitored files By default, a Logtail configuration on each server can be used to monitor a maximum of 10,000 files. The Logtail on each server can monitor a maximum of 100,000 files. Excessive files are not monitored.
If the upper limit is reached, you can perform the following operations:
  • Use more exact names to specify the monitored directories in each Logtail configuration.
  • Increase the value of the mem_usage_limit parameter to raise the threshold of available Logtail memory resources. For more information, see Set the startup parameters of Logtail.

    You can raise the threshold to no more than 2 GB. This way, the maximum number of files that can be monitored by using each Logtail configuration is increased to 100,000, and the maximum number of files that the Logtail on each server can monitor is increased to 1,000,000.

Default resources By default, Logtail can occupy up to 40% of the CPU and 256 MB of memory. If logs are generated at a high speed, you can change the values of the related parameters. For more information, see Set the startup parameters of Logtail.
Policy used to process excessive resource consumption If the amount of resources occupied by Logtail remains higher than the upper limit for more than 5 minutes, Logtail is forcibly restarted. The restart may cause data loss or duplication.
Table 5. Limits on troubleshooting
Item Description
Network error handling If a network error occurs, Logtail automatically retries the data collection task and adjusts the retry interval.
Processing of threshold-crossing events If a data transmission speed exceeds the upper limit of a Logstore, Logtail blocks log collection and automatically retries the data collection task.
Maximum retry period before timeout If data fails to be transmitted and the issue lasts for more than 6 hours, Logtail discards the data.
Status self-check If an exception occurs, Logtail restarts. For example, if an application unexpectedly exits or the resource usage exceeds the specified upper limit, Logtail restarts.
Table 6. Other limits
Item Description
Log collection latency In most cases, a latency of less than 1 second exists between the point in time at which a log is written to disk and the point in time at which Logtail collects the log. However, if the log collection is blocked, the latency increases.
Log upload policy Before Logtail uploads logs, Logtail aggregates the logs in the same file. Logtail starts to upload logs when the number of logs exceeds 2,000, the total size of logs exceeds 2 MB, or the log collection duration exceeds 3 seconds.