In the following scenarios, you need to configure the Logtail startup parameters.
- Memory usage is high due to a large number of log files to be collected. The meta information of each file must be maintained in memory, including the file signature, collection location, and file name.
- CPU usage is high due to heavy log data traffic.
- Traffic sent to Log Service is heavy due to large data volume.
- Syslogs and TCP data streams are collected.
File sample (which only shows partial configuration items)
"cpu_usage_limit" : 0.4,
"mem_usage_limit" : 100,
"max_bytes_per_sec" : 2097152,
"process_thread_count" : 1,
"send_request_concurrency" : 4,
"streamlog_open" : false,
"streamlog_pool_size_in_mb" : 50,
"streamlog_rcv_size_each_call" : 1024,
"streamlog_tcp_port" : 11111,
"buffer_file_num" : 25,
"buffer_file_size" : 20971520,
"buffer_file_path" : "",
|cpu_usage_limit||CPU usage threshold, double type and calculated per core.||For example, the value 0.4 indicates the CPU usage of Logtail is limited to 40% of single-core capacity. Logtail restarts automatically when the threshold is exceeded. In many cases, the single-core processing capability is about 24 MB/s in simple mode and about 12 MB/s in full mode.|
|mem_usage_limit||In-memory usage threshold, int type and measured in MBs.||For example, the value 100 indicates the memory usage of Logtail is limited to 100 MBs. Logtail restarts automatically when the threshold is exceeded. If you need to collect more than 1,000 distinct files, properl increase the threshold value.|
|max_bytes_per_sec||Traffic limit on the raw data sent by Logtail, int type and measured in bytes per second.||For example, the value 2,097,152 indicates the data transfer rate of Logtail is limited to 2 MB/s.|
|process_thread_count||Number of threads Logtail uses to write data to log files.||The default value is 1, which supports a write speed of 24 MB/s in simple mode and 12 MB/s in full mode. Adjust the threshold only when necessary.|
|send_request_concurrency||By default, Logtail sends data packets asynchronously. You can set a larger asynchronous concurrency value if the write TPS is large.||By default, four asynchronous concurrencies are available. You can calculate the concurrency quantity based on the condition that one concurrency supports 0.5–1 MB/s network throughout. The actual concurrency quantity varies with network delay.|
|streamlog_open||Syslog reception switch, bool type.||
|streamlog_pool_size_in_mb||Size of the cache storing received syslogs, measured in MBs.||This parameter indicates the size of the memory pool storing received syslogs. Logtail requests memory of the specified size at one time when launched. Set the pool size according to the machine’s memory size and your needs.|
|streamlog_rcv_size_each_call||Size of the cache Logtail uses when calling the Linux
||You can increase the value in the case of heavy syslog traffic. The recommended value range is 1,024 to 8,192 bytes.|
|streamlog_formats||Method of parsing received syslogs.|
|streamlog_tcp_port||TCP port through which Logtail receives syslogs.||By default, port 11111 is used.|
|buffer_file_num||When a network exception occurs or the write quota is exceeded, Logtail writes the logs that are parsed in real time to a local file (located in the installation directory) and then tries to resend the logs to Log Service after recovery. This parameter indicates the maximum number of cached files.||The default value is 25 for public cloud users.|
|buffer_file_size||Maximum number of bytes per cached file. The (buffer_file_num * buffer_file_size) indicates the maximum disk space available for storing cached files.||The default value is 20,971,520 (20 MBs).|
|buffer_file_path||Directory that stores cached files. After you modify this parameter, manually transfer the files named in the format of
||The default value is null, indicating the cached files are stored in the Logtail installation directory (
- The preceding table only lists the common startup parameters. If
ilogtail_config.jsonhas parameters not listed above, the default values are applied.
- Add or modify the values of configuration parameters according to your need. Any unused configuration parameters (for example, parameters related to the collection of syslog data streams) do not need to be added to
ilogtail_config.jsonaccording to your needs. Confirm the modified configuration is JSON compatible.
Check that the modified configurations are JSON compatible.
Restart Logtail to apply the modified configuration.