This topic describes how to configure the startup parameters of Logtail.

Scenarios

You can configure the startup parameters of Logtail in the following scenarios:
  • The log files to be collected consume large storage capacity.
  • The CPU usage is high due to a large amount of log data.
  • A large amount of log data needs to be transferred to Log Service.

Startup parameter file

  • File path

    /usr/local/ilogtail/ilogtail_config.json

  • File format

    JSON

  • Sample file (only partial parameters are provided)
    {
        ...
        "cpu_usage_limit" : 0.4,
        "mem_usage_limit" : 100,
        "max_bytes_per_sec" : 2097152,
        "process_thread_count" : 1,
        "send_request_concurrency" : 4,
        "buffer_file_num" : 25,
        "buffer_file_size" : 20971520,
        "buffer_file_path" : "",
        ...
    }

Set startup parameters

  1. Open the ilogtail_config.json file.
  2. Set startup parameters.
    • The following table lists some commonly used startup parameters. For the startup parameters that are not listed in the table, use the default values.
    • We recommend that you only add the parameters that you need to the ilogtail_config.json file.
    Parameter Type Description
    cpu_usage_limit double The threshold of CPU usage for Logtail. The threshold is calculated based on a single core. Default value: 2. The minimum value is 0.1. The maximum value is the number of CPU cores of the server on which Logtail is installed.

    For example, if you set the parameter to 0.4, the usage of a single-core CPU for Logtail is limited to 40%. Logtail restarts when the CPU usage exceeds 40%.

    In most cases, the processing capacity of a single core is about 24 MB/s in the simple mode and 12 MB/s in the full regex mode.

    mem_usage_limit int The threshold of the memory usage of Logtail. Unit: MB. Default value: 2048. The minimum value is 128. The maximum value is the size of the available memory.

    For example, if you set the parameter to 100, the memory usage is limited to 100 MB. Logtail restarts if the memory usage exceeds 100 MB.

    If you need to collect more than 1,000 log files, you can increase the threshold value.

    max_bytes_per_sec int The maximum speed at which Logtail sends raw data. Unit: bytes/s. Default value: 20971520. Valid values: 1024 to 52428800.

    For example, if you set the parameter to 2097152, the speed is limited to 2 MB/s.

    Note If you specify a speed larger than 20971520 bytes/s (20 MB/s), the speed is not limited.
    process_thread_count int The number of threads that Logtail uses to process data. Default value: 1. Valid values: 1 to 64.

    Each thread provides a write speed of 24 MB/s in the simple mode and 12 MB/s in the full regex mode. We recommend that you do not modify the default value.

    send_request_concurrency int The number of asynchronous concurrencies that Logtail uses to send data. Default value: 20. Valid values: 1 to 1000. By default, Logtail asynchronously sends data.

    If the write transactions per second (TPS) is high, you can set this parameter to a large value. Each concurrency can provide 0.5 MB/s to 1 MB/s network throughput. The number of concurrencies varies with the network latency.

    Note If the value of this parameter is too high, the concurrencies may occupy a large number of network ports. In this case, you need to adjust TCP parameters. .
    buffer_file_num int The maximum number of cached files. Default value: 25. Valid values: 1 to 100.

    If a network error occurs or the written data exceeds the specified quota, Logtail caches parsed logs to local files in the installation directory. After the network is recovered, Logtail retries to send the cached logs.

    buffer_file_size int The maximum size of a cached file. Unit: bytes. Default value: 20971520 (20 MB). Valid values: 1048576 to 104857600.

    The maximum disk space that cached files can occupy is calculated by multiplying the value of the buffer_file_size parameter with the value of the buffer_file_num parameter.

    buffer_file_path String The directory in which cached files are stored. The default value is null. The null value indicates that cached files are stored in the /usr/local/ilogtail directory where Logtail is installed.

    If you modify the parameter, you must move the cached files named logtail\_buffer\_file_ * to the directory that is specified by the parameter. Then, Logtail can read, send, and delete the cached files.

    bind_interface String The name of the NIC that is associated with the server. The default value is null. If you retain the default value, an available NIC is automatically associated with the server.

    If you set the parameter to eth1, Logtail uses the NIC named eth1 to upload logs.

    Note This parameter is valid only for Logtail that runs in Linux.
    check_point_filename String The path in which the checkpoint file of Logtail is stored. Default value: /tmp/logtail_check_point.

    We recommend that Docker users modify the path and mount the path to the host. Otherwise, when a Docker container is released, duplicate files are collected by Logtail due to the loss of checkpoint information. For example, you can set the check_point_filename parameter to /data/logtail/check_point.dat in the Docker container, and add -v /data/docker1/logtail:/data/logtail to the startup command. Then, the /data/docker1/logtail directory of the host is mounted to the /data/logtail directory of the Docker container.

    user_config_file_path String The path of the Logtail configuration file. By default, the Logtail configuration files are stored in the directory where the binary process is stored, and the file name is user_log_config.json.

    We recommend that Docker users modify the path and mount the path to the host. Otherwise, when a Docker container is released, duplicate files are collected by Logtail due to the loss of checkpoint information. For example, you can set the user_config_file_path parameter to /data/logtail/user_log_config.json in the Docker container, and add -v /data/docker1/logtail:/data/logtail to the startup command. Then, the /data/docker1/logtail directory of the host is mounted to the /data/logtail directory of the Docker container.

    discard_old_data bool Specifies whether to drop historical logs. Default value: true. The value true indicates that the logs generated 12 hours before the current time are dropped.
    working_ip String The server IP address that Logtail reports to Log Service. Default value: null. The value null indicates that Log Service automatically obtains the server IP address.
    working_hostname String The server hostname that Logtail reports to Log Service. Default value: null. The value null indicates that Log Service automatically obtains the hostname of the server.
    max_read_buffer_size long The maximum size of each log that is read. Default value: 524288 bytes (512 KB). If the size of a log exceeds 512 KB, modify the parameter value.
    oas_connect_timeout long The connection timeout period after Logtail sends requests to obtain the Logtail configuration or AccessKey pair. Unit: seconds. Default value: 5.

    If the network is instable, or the connection fails to be established, modify the value of this parameter.

    oas_request_timeout long The connection timeout period after Logtail sends requests to obtain the Logtail configuration or AccessKey pair. Unit: seconds. Default value: 10.

    If the network is instable, or the connection fails to be established, modify the value of this parameter.

  3. Restart Logtail to apply your settings.
    /etc/init.d/ilogtaild stop
    /etc/init.d/ilogtaild start
    /etc/init.d/ilogtaild status