This topic describes the limits of Logtail, including the limits on supported environments, log files, containers, checkpoints, Logtail configurations, machine groups, performance metrics, and troubleshooting.
Runtime environments
Limit | Description |
Architecture |
|
Computing resources |
The actual usage varies based on the collection rate, monitored directories, number of log files, and number of synchronously sent requests. Make sure that the actual usage is less than 80% of the limit. |
System environments | For supported operating systems, see System types. |
Kubernetes |
Important The Logtail components belong to the system-cluster-critical class. If cluster resources are insufficient, we recommend that you do not deploy the components. Otherwise, the existing pods on the node may be evicted. |
Docker | The collection of stdout and stderr from containers has the following limits:
|
Storage medium | We do not recommend that you use shared network storage media, such as NAS or OSS. Otherwise, data truncation, content inconsistency, or collection interruption may occur. We recommend that you use EBS. |
File collection
Limit | Description |
Size of a single log | By default, the maximum size of a log is 512 KB. You can use the max_read_buffer_size startup parameter to adjust the value. The maximum value is 8 MB. For more information, see Set Logtail startup parameters. If a multi-line log is split based on a regular expression that is used to match the beginning of the first line of a log, the maximum size of each log after splitting is still 512 KB. If the size of a log exceeds 512 KB, the log is forcefully split into multiple logs for collection. For example, if the size of a log is 1,025 KB, the log is split into logs of the following sizes: 512 KB, 512 KB, and 1 KB. Then, the logs are collected in sequence and considered incomplete logs. |
Log file encoding | Logtail supports log files that are encoded in UTF-8 and GBK. We recommend that you use UTF-8-encoded log files to improve processing performance. Warning If log files are encoded in other formats, issues such as garbled characters and data loss may occur. |
Size of a log file | Unlimited. |
Log file rotation | By default, the maximum number of log files in a rotation queue is 20. You can use the logreader_max_rotate_queue_size startup parameter to adjust the value. For more information, see Set Logtail startup parameters. You can set the collection path to Important Make sure that the two formats do not exist at the same time in a Logtail instance. If the two formats exist at the same time, the logs in a log file may be collected by using multiple Logtail configurations, and duplicate data may be collected. If more than 20 log files are not processed, new logs will be lost. In this case, you must check whether the write quota of shards exceeds the limit and adjust concurrency-related parameters. For more information, see Recommended parameter values. |
Log collection behavior performed when log parsing is blocked | When log parsing is blocked, Logtail keeps the descriptor of the log file open to prevent the log file from being deleted during the blocking period and log loss. If the log file is rotated multiple times during the blocking period, Logtail puts the log file into a rotation queue. |
Regular expression | Logtail uses regular expressions that are compatible with Perl. |
JSON | Logtail fully supports standard JSON (RFC7159 and ECMA-404). Logtail does not support non-standard JSON, such as |
Multiple Logtail configurations for one log file | By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect logs from a log file by using multiple Logtail configurations, see How do I collect logs from a log file by using multiple Logtail configurations? Important When you use multiple Logtail configurations to collect logs from a log file, the read I/O, computing resources, and network I/O increase accordingly. |
File opening behavior | Logtail keeps the log files from which you want to collect logs and the log files in a rotation queue open to ensure the integrity of collected data. A log file is closed in the following scenarios:
If you want to release the file handle within a controllable period of time after a file is deleted, regardless of whether the file is completely collected or logs are still being written to the file, you can use the force_release_deleted_file_fd_timeout startup parameter to set a timeout period. For more information, see Set Logtail startup parameters. |
First log collection behavior | Logtail collects data only from incremental log files. If the size of a log file exceeds the limit of 1 MB the first time the modification to the log file is detected, Logtail collects data from the last 1 MB. If the size of the log file does not exceed 1 MB, Logtail collects data from the beginning of the log file. The limit for container stdout and stderr is 512 KB. You can use the tail_size_kb parameter in a Logtail configuration to adjust the size of data that is collected from a new file for the first time. For more information, see Logtail configuration. If a log file is not modified after a Logtail configuration is delivered, Logtail does not collect data from the log file. If you want to collect historical logs, see Import historical log files. |
File overwriting behavior | Logtail uses an inode and the hash value of the first 1,024 bytes of a log file to identify the log file. If a log file is overwritten and the inode or the hash value of the first 1,024 bytes of the log file changes, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. If the inode or the hash value does not change, the logs in the log file are not collected. |
File transfer behavior | If a log file is transferred and the matched Logtail configuration is not used to collect logs from the log file before the log file is transferred, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. In this scenario, if the matched Logtail configuration is used to collect logs from the log file, the logs in the log file are not collected. |
File collection history | Logtail retains the historical collection progress of historical log files in the memory to ensure that only incremental data is collected after the log files are changed. If the historical collection progress of a historical log file is retained longer than a specified period and new data is written to the log file, duplicate data is collected.
|
Non-standard text logs | For log lines that contain |
Container collection
Both the limits on log files and the limits on containers apply when you use Logtail to collect container logs.
Limit | Description |
First log collection behavior | When you collect container stdout and stderr and if the size of a log file exceeds the limit of 512 KB the first time the modification to the log file is detected, Logtail collects data from the last 512 KB. If the size of the log file does not exceed 512 KB, Logtail collects data from the beginning of the log file. You can use the StartLogMaxOffset parameter in a Logtail configuration to adjust the value. For more information, see Use the console to collect container stdout and stderr in DaemonSet mode. |
Symbolic links | When you collect logs from container files, files and directories of files cannot be symbolic links. |
Container lifecycle | By default, Logtail can collect logs from a container only if the container lifecycle lasts 10 seconds or longer. When you collect logs from a container file, Logtail limits the number of log updates in the container file to 10 within a 3-minute period to ensure collection performance. You can use the docker_config_update_interval and max_docker_config_update_times startup parameters to adjust the values. For more information, see Set Logtail startup parameters. |
File rotation for stdout and stderr | The stdout and stderr files of a container are rotated by Docker or kubelet. The default size of a log file that is rotated by kubelet is 10 MB. The size of a log file that is rotated by Docker is set to 100 MB in ACK. If the rate at which a container outputs stdout exceeds 10 MB/s, the log file is quickly rotated. In this case, we recommend that you collect logs from container files or modify the containerLogMaxSize parameter of kubelet to prevent log loss. |
Logging driver for stdout and stderr | If you use Docker as the container runtime, you must add daemon.json |
CheckPoint management
Limit | Description |
Checkpoint timeout period | In default scenarios, if a log file is not modified within 30 days, the checkpoint of the log file is automatically deleted. If |
Checkpoint storage policy | Checkpoints are automatically stored at intervals of 15 minutes and at the point in time when Logtail exits. You can use the check_point_dump_interval startup parameter to adjust the interval. For more information, see Set Logtail startup parameters. |
Checkpoint storage path | The default checkpoint storage path is |
Handling during downtime | Checkpoints are saved at regular intervals. If downtime occurs, data collection resumes from the last completely saved checkpoint. This may cause duplicate data collection. You can change the checkpoint storage policy to prevent duplicate data collection. |
Logtail configurations
Limit | Description |
Latency for configuration updates to take effect | Updates to Logtail configurations that are performed by using the Simple Log Service console or by calling an API operation require approximately 30 seconds to take effect. |
Dynamic loading of Logtail configurations | Logtail configurations can be dynamically loaded. An update to a Logtail configuration does not affect other Logtail configurations. |
Number of Logtail configurations that can be dynamically loaded for a single Logtail instance | Unlimited. However, we recommend that you load no more than 100 Logtail configurations for a server. |
Log generation by using a third-party flusher | If a Logtail configuration is created in the Simple Log Service console or by calling an API operation, the Logtail configuration is associated with a Logstore. Therefore, when you configure a third-party flusher in your plug-in configuration, Logtail automatically sends a copy of data to the Logstore. |
Multi-account and cross-account | Yes, you can use Logtail to collect logs across Alibaba Cloud accounts. For more information, see Configure a user identifier and Use Logtail to collect container logs across Alibaba Cloud accounts. |
Multi-region | By default, multi-region collection is not supported. If you need this feature, submit a ticket. |
Global Accelerator | Yes, you can use Global Accelerator. After you enable global acceleration on the Simple Log Service side, you must configure global acceleration on the Logtail side. For more information, see Enable Global Accelerator. |
Machine groups
Limit | Description |
Number of machines | Unlimited. However, we recommend that you configure no more than 100,000 machines. Otherwise, heartbeats cannot be obtained. |
Number of Logtail configurations that can be applied | Unlimited. However, we recommend that you apply no more than 1,000 Logtail configurations. |
Performance
Limit | Description |
Throughput for log processing | The default transmission speed of raw logs is limited to 20 MB/s. Log data is uploaded after it is encoded and compressed. The compression ratio ranges from 5:1 to 10:1. If the log traffic exceeds the limit, logs may be lost. You can use the max_bytes_per_sec startup parameter to adjust the value. For more information, see Set Logtail startup parameters. |
Maximum processing speed for logs | Single-core-enabled processing speed:
You can use the process_thread_count startup parameter to set multiple processing threads. This can improve the performance by 1.5 to 3 times. |
Maximum numbers of monitored directories and files | The maximum numbers of monitored directories and files are related to the mem_usage_limit parameter. The default value of the parameter is 384 MB for hosts and 2,048 MB for containers. The maximum numbers of monitored directories and files are classified into the following categories:
If the number for one of the preceding categories reaches the upper limit, Logtail no longer monitors the rest of the directories and files that correspond to the category. You can increase the upper limit by improving the precision of the monitored directories in a Logtail configuration or by adjusting the mem_usage_limit startup parameter. For more information about the mem_usage_limit parameter, see Set Logtail startup parameters. Logtail installed on a Linux server allows you to use the inotify mechanism to monitor directories. This helps shorten the latency of log collection. The maximum number of directories, including subdirectories, which can be monitored by using the inotify mechanism is 3,000. |
Policy used to process excessive resource consumption | If the amount of resources occupied by Logtail remains higher than the upper limit for 5 minutes, Logtail is forcefully restarted. The restart may cause data loss or duplication. |
Multi-tenant data isolation | Logtail isolates data at the configuration level. If an exception occurs in a Logtail configuration, other Logtail configurations are not affected. |
Log collection latency | In normal cases, Logtail can collect a log less than 1 second after the log is written to disk. |
Log upload policy | Before Logtail uploads logs, Logtail aggregates the logs in the same file. Logtail starts to upload logs when the number of logs exceeds 4,000, the total size of logs exceeds 512 KB, or the log collection duration exceeds 3 seconds. If any of these conditions is met, Logtail uploads the logs. |
Error handling
Limit | Description |
Network error handling | If a network error occurs, Logtail automatically retries the data collection task and adjusts the retry interval. In extreme cases, logs may be repeatedly collected or discarded due to the following issues:
|
Processing of threshold-crossing events | If a data transmission speed exceeds the upper limit of a Logstore, Logtail blocks log collection and automatically retries the data collection task. We recommend that you increase the number of shards in the Logstore. |
Time errors of Logtail | If retries fail five times because the time difference between the request time and the response time is larger than 15 minutes, Logtail discards related data. A maximum of five retries are allowed. We recommend that you correct the time of your machine where Logtail resides. |
Non-existence of a specified project or Logstore | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. The failure may occur if you deleted your Logstore by calling an API operation. We recommend that you delete the Logtail configurations and disassociate the Logtail configurations from your machine groups by calling API operations. |
Authentication failure | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. The failure may occur in the following scenarios:
|
Other unknown errors | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. |
Maximum retry period before timeout | If data fails to be transmitted and the issue lasts for more than 6 hours, Logtail discards the data. |
Status self-check | If an exception occurs, Logtail restarts. For example, if an application unexpectedly exits or the resource usage exceeds the specified upper limit, Logtail restarts. |
Exceeding maximum numbers of monitored directories and files | Logtail cannot find the path to collect logs in a timely and accurate manner, which may lead to data loss. |
Significant collection delay | When the log collection progress falls behind the log generation progress and more than 20 unprocessed log rotations occur, data loss may occur. |