Logtail startup configuration parameters

Last Updated: Jul 05, 2017

Application scenarios

In the following scenarios, you need to configure the Logtail startup parameters.

  • Memory usage is high due to a large number of log files to be collected. The meta information of each file must be maintained in memory, including the file signature, collection location, and file name.
  • CPU usage is high due to heavy log data traffic.
  • Traffic sent to Log Service is heavy due to large data volume.
  • Syslogs and TCP data streams are collected.

Startup configuration

  • File path

    /usr/local/ilogtail/ilogtail_config.json

  • File format

    JSON

  • File sample (which only shows partial configuration items)

    1. {
    2. ...
    3. "cpu_usage_limit" : 0.4,
    4. "mem_usage_limit" : 100,
    5. "max_bytes_per_sec" : 2097152,
    6. "process_thread_count" : 1,
    7. "send_request_concurrency" : 4,
    8. "streamlog_open" : false,
    9. "streamlog_pool_size_in_mb" : 50,
    10. "streamlog_rcv_size_each_call" : 1024,
    11. "streamlog_formats":[],
    12. "streamlog_tcp_port" : 11111,
    13. "buffer_file_num" : 25,
    14. "buffer_file_size" : 20971520,
    15. "buffer_file_path" : "",
    16. ...
    17. }

Common configuration parameters

Parameter Value Description
cpu_usage_limit CPU usage threshold, double type and calculated per core. For example, the value 0.4 indicates the CPU usage of Logtail is limited to 40% of single-core capacity. Logtail restarts automatically when the threshold is exceeded. In many cases, the single-core processing capability is about 24 MB/s in simple mode and about 12 MB/s in full mode.
mem_usage_limit In-memory usage threshold, int type and measured in MBs. For example, the value 100 indicates the memory usage of Logtail is limited to 100 MBs. Logtail restarts automatically when the threshold is exceeded. If you need to collect more than 1,000 distinct files, properl increase the threshold value.
max_bytes_per_sec Traffic limit on the raw data sent by Logtail, int type and measured in bytes per second. For example, the value 2,097,152 indicates the data transfer rate of Logtail is limited to 2 MB/s.
process_thread_count Number of threads Logtail uses to write data to log files. The default value is 1, which supports a write speed of 24 MB/s in simple mode and 12 MB/s in full mode. Adjust the threshold only when necessary.
send_request_concurrency By default, Logtail sends data packets asynchronously. You can set a larger asynchronous concurrency value if the write TPS is large. By default, four asynchronous concurrencies are available. You can calculate the concurrency quantity based on the condition that one concurrency supports 0.5–1 MB/s network throughout. The actual concurrency quantity varies with network delay.
streamlog_open Syslog reception switch, bool type. false indicates that syslog reception is disabled; true indicates that syslog reception is enabled.
streamlog_pool_size_in_mb Size of the cache storing received syslogs, measured in MBs. This parameter indicates the size of the memory pool storing received syslogs. Logtail requests memory of the specified size at one time when launched. Set the pool size according to the machine’s memory size and your needs.
streamlog_rcv_size_each_call Size of the cache Logtail uses when calling the Linux socket rcv interface, measured in bytes. You can increase the value in the case of heavy syslog traffic. The recommended value range is 1,024 to 8,192 bytes.
streamlog_formats Method of parsing received syslogs.
streamlog_tcp_port TCP port through which Logtail receives syslogs. By default, port 11111 is used.
buffer_file_num When a network exception occurs or the write quota is exceeded, Logtail writes the logs that are parsed in real time to a local file (located in the installation directory) and then tries to resend the logs to Log Service after recovery. This parameter indicates the maximum number of cached files. The default value is 25 for public cloud users.
buffer_file_size Maximum number of bytes per cached file. The (buffer_file_num * buffer_file_size) indicates the maximum disk space available for storing cached files. The default value is 20,971,520 (20 MBs).
buffer_file_path Directory that stores cached files. After you modify this parameter, manually transfer the files named in the format of logtail\_buffer\_file_* in the old cache directory to the new directory so that Logtail can read the cached files and delete them after sending. The default value is null, indicating the cached files are stored in the Logtail installation directory (/usr/local/ilogtail).

Note:

  • The preceding table only lists the common startup parameters. If ilogtail_config.json has parameters not listed above, the default values are applied.
  • Add or modify the values of configuration parameters according to your need. Any unused configuration parameters (for example, parameters related to the collection of syslog data streams) do not need to be added to ilogtail_config.json.

Modify configuration

  1. Configure ilogtail_config.json according to your needs. Confirm the modified configuration is JSON compatible.

    Check that the modified configurations are JSON compatible.

  2. Restart Logtail to apply the modified configuration.

    1. /etc/init.d/ilogtaild stop
    2. /etc/init.d/ilogtaild start
    3. /etc/init.d/ilogtaild status
Thank you! We've received your feedback.