Log Service limits the collection performance of Logtail to prevent Logtail from consuming excessive server resources. If Logtail consumes excessive server resources, other services on the server may be affected. If you want to improve the collection performance of Logtail, you can modify the startup parameters of Logtail.

Scenarios

You can modify the startup parameters of Logtail in the following scenarios:
  • You need to collect logs from a large number of log files and the log files occupy a large amount of memory. For example, you need to collect logs from more than 100 files or the log monitoring directory contains more than 5,000 log files.
  • Log data is transmitted at a high speed, which causes high CPU utilization. For example, Logtail collects log data at a speed that exceeds 2 MB/s in simple mode and at a speed that exceeds 1 MB/s in full regex mode.
  • Logtail sends data to Log Service at a speed that exceeds 10 MB/s.

Recommended parameter values

If you want to collect logs from JSON files, you can use the following parameter values that are obtained from real-world practice. The collection performance of Logtail in full regex mode and in delimiter mode is similar to the collection performance of Logtail in JSON mode. The collection performance of Logtail in simple mode is five times higher than the collection performance of Logtail in JSON mode. Both the complexity of data and rules and the numbers of directories and files from which you want to collect logs affect CPU utilization and memory usage. We recommend that you configure the following parameters based on the values in the table and your business requirements.

  • Host environment
    Parameter Default collection speed Collection speed higher than 10 MB/s Collection speed higher than 20 MB/s Collection speed higher than 40 MB/s
    cpu_usage_limit 0.4 1 2 4
    mem_usage_limit 384 1024 2048 4096
    max-bytes-per-sec 20971520 209715200 209715200 209715200
    process_thread_count 1 2 4 8
    send_request_concurrency 4 20 40 80
  • Container or Kubernetes environment
    Environment variable Default collection speed Collection speed higher than 10 MB/s Collection speed higher than 20 MB/s Collection speed higher than 40 MB/s
    cpu_usage_limit 2 3 5 9
    mem_usage_limit 512 1024 2048 4096
    max_bytes_per_sec 209715200 209715200 209715200 209715200
    process_thread_count 1 2 4 8
    send_request_concurrency 20 20 40 80
    resources.limits.cpu 500M 1000M 2000M 4000M
    resources.limits.memory 1 Gi 2 Gi 3 Gi 5 Gi

    If you want to collect logs from a container or a Kubernetes cluster, you can modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration. You can also modify resources.limits.cpu and resources.limits.memory in daemonset > kube-system > logtail-ds to prevent the excessive usage of container resources.

If you configure the Logtail startup parameters based on the values of the Collection speed higher than 40 MB/s column in the preceding tables, the collection performance of Logtail approaches the upper limit. In this case, the performance does not significantly improve even if more threads are created. The following table describes the upper limit of the collection performance that Logtail can deliver in different collection modes.

Note The actual collection performance may vary based on the test environment and the production environment.
Collection mode Upper limit
Simple mode 440 MB/s
Full regex mode 70 MB/s
Delimiter mode 75 MB/s
JSON mode 75 MB/s

Configure the startup parameters of Logtail

  1. Open the /usr/local/ilogtail/ilogtail_config.json file on the server on which Logtail is installed.

    If you want to collect logs from a host, you can perform this step to configure the startup parameters of Logtail.

    If you want to collect logs from a container or a Kubernetes cluster, you can modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration.

  2. Configure the startup parameters of Logtail based on your business requirements.

    The following example shows the startup parameters of Logtail:

    {
        ...
        "cpu_usage_limit" : 0.4,
        "mem_usage_limit" : 384,
        "max_bytes_per_sec" : 20971520,
        "process_thread_count" : 1,
        "send_request_concurrency" : 4,
        "buffer_file_num" : 25,
        "buffer_file_size" : 20971520,
        "buffer_file_path" : "",
        ...
    }
    Note
    • The following table describes the commonly used startup parameters of Logtail. You can retain the default values for other startup parameters.
    • You can add or modify startup parameters based on your business requirements.
    Table 1. Startup parameters of Logtail
    Parameter Type Description Example
    cpu_usage_limit double The CPU utilization threshold for Logtail. The calculation is based on a single core.
    • Valid values: 0.1 to the number of CPU cores of the current server
    • Default value: 2
    Warning cpu_usage_limit specifies a soft limit. The actual CPU utilization of Logtail may exceed the limit. If the CPU utilization of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts.

    For example, you set the parameter to 0.4. If the CPU utilization of Logtail remains higher than 40% for 5 minutes based on a single core, Logtail automatically restarts.

    In most cases, a single core supports a collection speed of 100 MB/s in simple mode and 20 MB/s in full regex mode.

    "cpu_usage_limit" : 0.4
    mem_usage_limit int The memory usage threshold for Logtail. Unit: MB.
    • Valid values: 128 to 2048
    • Default value: 384
    Warning mem_usage_limit specifies a soft limit. The actual memory usage of Logtail may exceed the limit. If the memory usage of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts.
    The following list describes how the setting of mem_usage_limit affects the number of files that can be monitored:
    • If you use the default value, each Logtail configuration on a server allows Logtail to monitor up to 19,200 files. Logtail on a server can monitor up to 192,000 files.
    • If you set the value to 2048, each Logtail configuration on a server allows Logtail to monitor up to 100,000 files. Logtail on a server can monitor up to 1,000,000 files.
    The maximum number of files that can be monitored is calculated by using the following formulas:
    • Maximum number of files that can be monitored by Logtail for a Logtail configuration = Value of mem_usage_limit/100 × 5,000
    • Maximum number of files that can be monitored by Logtail on a server = Value of mem_usage_limit/100 × 50,000
    "mem_usage_limit" : 384
    max_bytes_per_sec int The highest speed at which Logtail sends raw data. Unit: bytes per second.
    • Valid values: 1024 to 52428800
    • Default value: 20971520

    For example, if you set the parameter to 2097152, the highest speed at which Logtail sends data is 2 MB/s.

    Notice If you set the parameter to a value that is greater than 20971520, the speed at which Logtail sends data is not limited. The value 20971520 indicates that the speed is 20 MB/s.
    "max_bytes_per_sec" : 2097152
    process_thread_count int The number of threads that are used by Logtail to process data.
    • Valid values: 1 to 64
    • Default value: 1

    In most cases, a thread provides a write speed of 24 MB/s in simple mode and 12 MB/s in full regex mode. We recommend that you retain the default value for this parameter.

    "process_thread_count" : 1
    send_request_concurrency int The maximum number of concurrent requests that can be sent by Logtail to asynchronously send data.
    • Valid values: 1 to 1000
    • Default value: 20

    If Log Service provides a high transactions per second (TPS), you can set this parameter to a larger value. Each concurrent request supports a network throughput of 0.5 MB/s to 1 MB/s. The actual network throughput for a concurrent request varies based on the network latency.

    Note If the value of this parameter is large, concurrent requests may occupy an excessive number of network ports. In this case, you must adjust the TCP-related parameters.
    "send_request_concurrency" : 4
    buffer_file_num int The maximum number of files that can be cached.
    • Valid values: 1 to 100
    • Default value: 25

    If a network error occurs or the limits of data write are reached, Logtail caches parsed logs to the local files in the installation directory. Logtail parses raw logs in real time. After the issues are fixed, Logtail retries to send the cached logs.

    "buffer_file_num" : 25
    buffer_file_size int The maximum size of a cached file. Unit: bytes.
    • Valid values: 1048576 to 104857600
    • Default value: 20971520

    The maximum disk space that can be occupied by cached files is calculated by multiplying the value of the buffer_file_size parameter by the value of the buffer_file_num parameter.

    "buffer_file_size" : 20971520
    buffer_file_path String The directory in which cached files are stored. This parameter is empty by default, which indicates that cached files are stored in the installation directory of Logtail. The default directory is /usr/local/ilogtail.

    If you specify a value for this parameter, you must move the cached files whose name matches logtail\_buffer\_file_* from the installation directory of Logtail to the directory that you specify. This way, Logtail can read, send, and then delete the cached files.

    "buffer_file_path" : ""
    bind_interface String The name of the network interface controller (NIC) that is associated with the server on which Logtail is installed. This parameter is empty by default, which indicates that the server is automatically associated with an available NIC.

    If you specify a value for this parameter, such as eth1, Logtail uses the NIC to upload logs.

    This parameter is available only if Logtail runs on a Linux server.

    "bind_interface" : ""
    check_point_filename String The path to the checkpoint files of Logtail. Default value: /tmp/logtail_check_point. "check_point_filename" : /tmp/logtail_check_point
    check_point_dump_interval int The interval at which Logtail updates checkpoint files. Default value: 900. Unit: seconds. If you retain the default value, Logtail updates checkpoint files at 15-minute intervals.

    This parameter is available only for Logtail V1.0.19 or later.

    "check_point_dump_interval" : 900
    user_config_file_path String The path to the file that stores Logtail configurations. The file is named user_log_config.json and stored in the directory of the BIN file that is created for the Logtail process. "user_config_file_path" : user_log_config.json
    docker_file_cache_path String The path to the file that records the path mappings between container files and host files. By default, the path is /usr/local/ilogtail/docker_path_config.json.

    This parameter is available only for Logtail V0.16.54 or later.

    "docker_file_cache_path": /usr/local/ilogtail/docker_path_config.json
    discard_old_data Boolean Specifies whether to discard historical logs. Default value: true. This value indicates that logs that were generated more than 12 hours before the current time are discarded. "discard_old_data" : true
    ilogtail_discard_interval int The time threshold for discarding logs. If the difference between the time at which the logs were generated and the current time exceeds the threshold, the logs are discarded. Default value: 43200. Unit: seconds. The value 43200 indicates that the threshold is 12 hours. "ilogtail_discard_interval": 43200
    working_ip String The server IP address that is reported by Logtail to Log Service. This parameter is empty by default, which indicates that Log Service automatically obtains the IP address of the server on which Logtail is installed. "working_ip" : ""
    working_hostname String The server hostname that is reported by Logtail to Log Service. This parameter is empty by default, which indicates that Log Service automatically obtains the hostname of the server on which Logtail is installed. "working_hostname" : ""
    max_read_buffer_size long The maximum size of a log that Logtail can read. Unit: bytes. Default value: 524288. The default value 524288 indicates that the maximum size is 512 KB. Maximum value: 4194304. The value 4194304 indicates that the maximum size is 4 MB.

    If the size of a log exceeds 524,288 bytes, you can change the value of this parameter.

    "max_read_buffer_size" : 524288
    oas_connect_timeout long The timeout period of the connection that is established by Logtail to send a request to obtain the Logtail configuration or AccessKey pair. Default value: 5. Unit: seconds.

    If the connections cannot be established before timeout due to poor network conditions, you can change the value of this parameter.

    "oas_connect_timeout" : 5
    oas_request_timeout long The timeout period of the request that is sent by Logtail to obtain the Logtail configuration or AccessKey pair. Default value: 10. Unit: seconds.

    If the connections cannot be established before timeout due to poor network conditions, you can change the value of this parameter.

    "" : 10
    data_server_port long If you set the data_server_port parameter to 443, Logtail transfers data to Log Service over HTTPS.

    This parameter is available only for Logtail V1.0.10 or later.

    "data_server_port": 443
    enable_log_time_auto_adjust Boolean If you set the enable_log_time_auto_adjust parameter to true, the log time is adapted to the local time of the server.

    To ensure data security, Log Service checks the time information in requests, including the requests sent by Logtail. This information indicates the time at which a request is sent. Log Service rejects requests that are sent 15 minutes earlier or later than the time in Log Service. The time information in a request is considered as the local time of the server. In some test scenarios, the local time must be changed to a future point in time. If you change the local time of the server, Log Service rejects the requests from Logtail, and data cannot be written to Log Service. You can use this parameter to adapt the log time to the local time of the server.

    This parameter is available only for Logtail V1.0.19 or later.

    Notice
    • If you set the enable_log_time_auto_adjust parameter to true, the offset between the time in Log Service and the local time of the server is added to the log time. The offset is updated only if a request is rejected by Log Service. Therefore, the time of a log that is queried by Log Service may be different from the time at which the log is written.
    • Part of the logic for Logtail changes based on the incremental increase of the system time. We recommend that you restart Logtail after you change the local time of the server.
    "enable_log_time_auto_adjust": true
    accept_multi_config Boolean Specifies whether to allow Logtail to collect data from the same file by using multiple Logtail configurations. Default value: false. This value indicates that Logtail cannot collect data from the same file by using multiple Logtail configurations.

    By default, Logtail can use only one Logtail configuration to collect data from a file. If you want to allow Logtail to collect data from a file by using multiple Logtail configurations, you can set this parameter to true. Each Logtail configuration has an independent collection process. If multiple Logtail configurations are used to collect data from the same file, the CPU utilization and memory usage increase.

    This parameter is available only for Logtail V0.16.26 or later.

    "accept_multi_config": true
    enable_checkpoint_sync_write Boolean Specifies whether to enable the sync write feature. Default value: false. This value indicates that the sync write feature is disabled.

    The sync write feature is used together with the ExactlyOnce write feature. After you enable the ExactlyOnce write feature, Logtail records fine-grained checkpoints by file to the disk of the server on which Logtail is installed. By default, Logtail does not call the sync function to write checkpoints to the disk. However, if buffered data fails to be written to the disk when the server restarts, the checkpoints may be lost. In this case, you can set the enable_checkpoint_sync_write parameter to true to enable the sync write feature. For more information, see Additional information: ExactlyOnce write feature.

    This parameter is available only for Logtail V1.0.20 or later.

    "enable_checkpoint_sync_write": false
    enable_env_ref_in_config Boolean Specifies whether to enable the environment variable replacement feature in Logtail configurations. Default value: false.

    After this feature is enabled, you can use ${xxx} as the placeholder for the environment variable xxx when you create a Logtail configuration in the Log Service console. For example, if you set Log Path to /${xxx}/logs and the environment variable to xxx=root, the path to the file from which Logtail collects logs is /root/logs.

    If ${ and } are used in your Logtail configuration, you can use $${ and $} to escape the characters.

    This parameter is available only for Logtail V1.0.31 or later.

    "enable_env_ref_in_config": false
    docker_config_update_interval int The minimum interval at which the container path is updated. Default value for versions earlier than Logtail V1.0.32: 10. Default value for Logtail V1.0.32 or later: 3. Unit: seconds.

    This parameter is used together with the max_docker_config_update_times parameter. If one of the values for the two parameters is reached, the container path is no longer updated.

    "docker_config_update_interval": 3
    max_docker_config_update_times int The maximum number of times that the container path can be updated within 3 minutes. Default value for versions earlier than Logtail V1.0.32: 3. Default value for Logtail V1.0.32 or later: 10. By default, if the container path is updated more than three times within a 3-minute period, the container path cannot be updated again until 3 minutes later. "max_docker_config_update_times": 10
  3. Restart Logtail for the new settings to take effect.
    /etc/init.d/ilogtaild stop && /etc/init.d/ilogtaild start                        

    After you restart Logtail, you can run the /etc/init.d/ilogtaild status command to check the status of Logtail.

Appendix: Environment variables

The following table describes the mappings between environment variables and the startup parameters of Logtail. For information about the startup parameters of Logtail, see Table 1.

Table 2. Mappings between environment variables and the startup parameters of Logtail
Parameter Environment variable Priority Supported version
cpu_usage_limit cpu_usage_limit If you use environment variables and the configuration file named /usr/local/ilogtail/ilogtail_config.json to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.32 or later
mem_usage_limit mem_usage_limit If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.32 or later
max_bytes_per_sec max_bytes_per_sec If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.32 or later
process_thread_count process_thread_count If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.32 or later
send_request_concurrency send_request_concurrency If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.32 or later
check_point_filename check_point_filename or ALIYUN_LOGTAIL_CHECK_POINT_PATH If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the environment variables take effect. Logtail V0.16.36 or later
docker_file_cache_path docker_file_cache_path If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.54 or later
user_config_file_path user_config_file_path If you use environment variables and the configuration file of Logtail to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
discard_old_data discard_old_data If you use environment variables and the configuration file of Logtail to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
working_ip working_ip or ALIYUN_LOGTAIL_WORKING_IP If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
working_hostname working_hostname or ALIYUN_LOGTAIL_WORKING_HOSTNAME If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
max_read_buffer_size max_read_buffer_size If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
oas_connect_timeout oas_connect_timeout If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
oas_request_timeout oas_request_timeout If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
data_server_port data_server_port If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
accept_multi_config accept_multi_config If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V0.16.56 or later
enable_log_time_auto_adjust enable_log_time_auto_adjust If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V1.0.19 or later
check_point_dump_interval check_point_dump_interval If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V1.0.19 or later
enable_checkpoint_sync_write enable_checkpoint_sync_write If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V1.0.20 or later
docker_config_update_interval docker_config_update_interval or ALIYUN_LOGTAIL_DOCKER_CONFIG_UPDATE_INTERVAL If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V1.0.29 or later
max_docker_config_update_times max_docker_config_update_times or ALIYUN_LOGTAIL_MAX_DOCKER_CONFIG_UPDATE_TIMES If you use environment variables and the configuration file to modify the startup parameters of Logtail, the modifications from the configuration file take effect. Logtail V1.0.29 or later