Log Service:Configure the startup parameters of Logtail

Scenarios

Recommended parameter values

If you want to collect logs from JSON files, you can use the following parameter values that are obtained from real-world practice. The collection performance of Logtail in full regex mode and in delimiter mode is similar to the collection performance of Logtail in JSON mode. The collection performance of Logtail in simple mode is five times higher than the collection performance of Logtail in JSON mode. Both the complexity of data and rules and the numbers of directories and files from which you want to collect logs affect CPU utilization and memory usage. We recommend that you configure the following parameters based on the values in the table and your business requirements.

• Host environment

  Parameter	Default collection speed	Collection speed higher than 10 MB/s	Collection speed higher than 20 MB/s	Collection speed higher than 40 MB/s
  cpu_usage_limit	0.4	1	2	4
  mem_usage_limit	384	1024	2048	4096
  max_bytes_per_sec	20971520	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	4	20	40	80

• Container or Kubernetes environment

  Environment variable	Default collection speed	Collection speed higher than 10 MB/s	Collection speed higher than 20 MB/s	Collection speed higher than 40 MB/s
  cpu_usage_limit	2	3	5	9
  mem_usage_limit	2048	2048	2048	4096
  max_bytes_per_sec	209715200	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	20	20	40	80
  resources.limits.cpu	500M	1000M	2000M	4000M
  resources.limits.memory	2 Gi	2 Gi	3 Gi	5 Gi

  If you want to collect logs from a container or a Kubernetes cluster, you can modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration. You can also modify resources.limits.cpu and resources.limits.memory in daemonset > kube-system > logtail-ds to prevent the excessive usage of container resources.

If you configure the Logtail startup parameters based on the values of the Collection speed higher than 40 MB/s column in the preceding tables, the collection performance of Logtail approaches the upper limit. In this case, the performance does not significantly improve even if more threads are created. The following table describes the upper limit of the collection performance that Logtail can deliver in different collection modes.

Configure the startup parameters of Logtail

1. Open the /usr/local/ilogtail/ilogtail_config.json file on the server on which Logtail is installed.

   If you want to collect logs from a host, you can perform this step to configure the startup parameters of Logtail.

   If you want to collect logs from a container or a Kubernetes cluster, you can modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration.

2. Configure the startup parameters of Logtail based on your business requirements.

   The following example shows the startup parameters of Logtail:

   {
       ...
       "cpu_usage_limit" : 0.4,
       "mem_usage_limit" : 384,
       "max_bytes_per_sec" : 20971520,
       "process_thread_count" : 1,
       "send_request_concurrency" : 4,
       "buffer_file_num" : 25,
       "buffer_file_size" : 20971520,
       "buffer_file_path" : "",
       ...
   }

   Note

   • The following table describes the commonly used startup parameters of Logtail. You can retain the default values for other startup parameters.
   • You can add or modify startup parameters based on your business requirements.

   Table 1. Startup parameters of Logtail

   Parameter	Type	Description	Example
   cpu_usage_limit	double	The CPU utilization threshold for Logtail. The calculation is based on a single core. Valid values: 0.1 to the number of CPU cores of the current server Default value: 0.4 Warning cpu_usage_limit specifies a soft limit. The actual CPU utilization of Logtail may exceed the limit. If the CPU utilization of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts. For example, you set the parameter to 0.4. If the CPU utilization of Logtail remains higher than 40% for 5 minutes based on a single core, Logtail automatically restarts. In most cases, a single core supports a collection speed of about 100 MB/s in simple mode and about 20 MB/s in full regex mode.	"cpu_usage_limit" : 0.4
   mem_usage_limit	int	The memory usage threshold for Logtail. Unit: MB. Valid values: 128 to 8192 Default value: 384 for a host environment and 2048 for Container Service for Kubernetes (ACK) components Warning mem_usage_limit specifies a soft limit. The actual memory usage of Logtail may exceed the limit. If the memory usage of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts. The collection speed, monitored directories, number of log files, and number of synchronously sent requests are related to the mem_usage_limit parameter. For more information, see Logtail limits.	"mem_usage_limit" : 384
   max_bytes_per_sec	int	The highest speed at which Logtail sends raw data. Unit: bytes per second. Valid values: 1024 to 52428800 Default value: 20971520 For example, if you set the parameter to 2097152, the highest speed at which Logtail sends data is 2 MB/s. Important If you set the parameter to a value that is greater than 20971520, the speed at which Logtail sends data is not limited. The value 20971520 indicates that the speed is 20 MB/s.	"max_bytes_per_sec" : 2097152
   process_thread_count	int	The number of threads that are used by Logtail to process data. Valid values: 1 to 64 Default value: 1 In most cases, a thread provides a write speed of 24 MB/s in simple mode and 12 MB/s in full regex mode. We recommend that you retain the default value for this parameter.	"process_thread_count" : 1
   send_request_concurrency	int	The maximum number of concurrent requests that can be sent by Logtail to asynchronously send data. Valid values: 1 to 1000 Default value: 20 If Log Service provides a high transactions per second (TPS), you can set this parameter to a larger value. Each concurrent request supports a network throughput of 0.5 MB/s to 1 MB/s. The actual network throughput for a concurrent request varies based on the network latency. Note If the value of this parameter is large, concurrent requests may use an excessive number of network ports. In this case, you must change the values of the TCP-related parameters.	"send_request_concurrency" : 4
   buffer_file_num	int	The maximum number of files that can be cached. Valid values: 1 to 100. Default value: 25 If a network error occurs or the limits of data write are reached, Logtail caches parsed logs to the local files in the installation directory. Logtail parses raw logs in real time. After the issues are fixed, Logtail retries to send the cached logs.	"buffer_file_num" : 25
   buffer_file_size	int	The maximum size of a cached file. Unit: bytes. Valid values: 1048576 to 104857600 Default value: 20971520 The maximum disk space that can be occupied by cached files is calculated by multiplying the value of the buffer_file_size parameter by the value of the buffer_file_num parameter.	"buffer_file_size" : 20971520
   buffer_file_path	String	The directory in which cached files are stored. This parameter is empty by default, which indicates that cached files are stored in the installation directory of Logtail. The default directory is /usr/local/ilogtail. If you specify a value for this parameter, you must move the cached files whose name matches logtail\_buffer\_file_* from the installation directory of Logtail to the directory that you specify. This way, Logtail can read, send, and then delete the cached files.	"buffer_file_path" : ""
   bind_interface	String	The name of the network interface controller (NIC) that is associated with the server on which Logtail is installed. This parameter is empty by default, which indicates that the server is automatically associated with an available NIC. If you specify a value for this parameter, such as eth1, Logtail uses the NIC to upload logs. This parameter is available only if Logtail runs on a Linux server.	"bind_interface" : ""
   check_point_filename	String	The path to the checkpoint files of Logtail. Default value: /tmp/logtail_check_point.	"check_point_filename" : /tmp/logtail_check_point
   check_point_dump_interval	int	The interval at which Logtail updates checkpoint files. Default value: 900. Unit: seconds. If you retain the default value, Logtail updates checkpoint files at 15-minute intervals. This parameter is available only for Logtail V1.0.19 or later.	"check_point_dump_interval" : 900
   user_config_file_path	String	The path to the file that stores Logtail configurations. The file is named user_log_config.json and stored in the directory of the BIN file that is created for the Logtail process.	"user_config_file_path" : user_log_config.json
   docker_file_cache_path	String	The path to the file that records the path mappings between container files and host files. By default, the path is /usr/local/ilogtail/docker_path_config.json. This parameter is available only for Logtail V0.16.54 or later.	"docker_file_cache_path": /usr/local/ilogtail/docker_path_config.json
   discard_old_data	Boolean	Specifies whether to discard historical logs. Default value: true. This value indicates that logs that were generated more than 12 hours before the current time are discarded.	"discard_old_data" : true
   ilogtail_discard_interval	int	The time threshold for discarding logs. If the difference between the time at which the logs were generated and the current time exceeds the threshold, the logs are discarded. Default value: 43200. Unit: seconds. The value 43200 indicates that the threshold is 12 hours.	"ilogtail_discard_interval": 43200
   working_ip	String	The server IP address that is reported by Logtail to Log Service. This parameter is empty by default, which indicates that Log Service automatically obtains the IP address of the server on which Logtail is installed.	"working_ip" : ""
   working_hostname	String	The server hostname that is reported by Logtail to Log Service. This parameter is empty by default, which indicates that Log Service automatically obtains the hostname of the server on which Logtail is installed.	"working_hostname" : ""
   max_read_buffer_size	long	The maximum size of a log that Logtail can read. Unit: bytes. Default value: 524288. The default value 524288 indicates that the maximum size is 512 KB. Maximum value: 8388608. The value 8388608 indicates that the maximum size is 8 MB. If the size of a log exceeds 524,288 bytes, you can change the value of this parameter.	"max_read_buffer_size" : 524288
   oas_connect_timeout	long	The timeout period of the connection that is established by Logtail to send a request to obtain the Logtail configuration or AccessKey pair. Default value: 5. Unit: seconds. If the connections cannot be established before timeout due to poor network conditions, you can change the value of this parameter.	"oas_connect_timeout" : 5
   oas_request_timeout	long	The timeout period of the request that is sent by Logtail to obtain the Logtail configuration or AccessKey pair. Default value: 10. Unit: seconds. If the connections cannot be established before timeout due to poor network conditions, you can change the value of this parameter.	"" : 10
   data_server_port	long	If you set the data_server_port parameter to 443, Logtail transfers data to Log Service over HTTPS. This parameter is available only for Logtail V1.0.10 or later.	"data_server_port": 443
   enable_log_time_auto_adjust	Boolean	If you set the enable_log_time_auto_adjust parameter to true, the log time is adapted to the local time of the server. To ensure data security, Log Service checks the time information in requests, including the requests sent by Logtail. This information indicates the time at which a request is sent. Log Service rejects requests that are sent 15 minutes earlier or later than the time in Log Service. The time information in a request is considered as the local time of the server. In some test scenarios, the local time must be changed to a future point in time. If you change the local time of the server, Log Service rejects the requests from Logtail, and data cannot be written to Log Service. You can use this parameter to adapt the log time to the local time of the server. This parameter is available only for Logtail V1.0.19 or later. Important If you set the enable_log_time_auto_adjust parameter to true, the offset between the time in Log Service and the local time of the server is added to the log time. The offset is updated only if a request is rejected by Log Service. Therefore, the time of a log that is queried by Log Service may be different from the time at which the log is written. Part of the logic for Logtail changes based on the incremental increase of the system time. We recommend that you restart Logtail after you change the local time of the server.	"enable_log_time_auto_adjust": true
   accept_multi_config	Boolean	Specifies whether to allow Logtail to collect data from the same file by using multiple Logtail configurations. Default value: false. This value indicates that Logtail cannot collect data from the same file by using multiple Logtail configurations. By default, Logtail can use only one Logtail configuration to collect data from a file. If you want to allow Logtail to collect data from a file by using multiple Logtail configurations, you can set this parameter to true. Each Logtail configuration has an independent collection process. If multiple Logtail configurations are used to collect data from the same file, the CPU utilization and memory usage increase. This parameter is available only for Logtail V0.16.26 or later.	"accept_multi_config": true
   enable_checkpoint_sync_write	Boolean	Specifies whether to enable the sync write feature. Default value: false. This value indicates that the sync write feature is disabled. The sync write feature is used together with the ExactlyOnce write feature. After you enable the ExactlyOnce write feature, Logtail records fine-grained checkpoints by file to the disk of the server on which Logtail is installed. By default, Logtail does not call the sync function to write checkpoints to the disk. However, if buffered data fails to be written to the disk when the server restarts, the checkpoints may be lost. In this case, you can set the enable_checkpoint_sync_write parameter to true to enable the sync write feature. For more information, see Logtail configurations. This parameter is available only for Logtail V1.0.20 or later.	"enable_checkpoint_sync_write": false
   enable_env_ref_in_config	Boolean	Specifies whether to enable the environment variable replacement feature in Logtail configurations. Default value: false. After this feature is enabled, you can use ${xxx} as the placeholder for the environment variable xxx when you create a Logtail configuration in the Log Service console. For example, if you set Log Path to /${xxx}/logs and the environment variable to xxx=user, the path to the file from which Logtail collects logs is /user/logs. If ${ and } are used in your Logtail configuration, you can use $${ and $} to escape the characters. This parameter is available only for Logtail V1.0.31 or later.	"enable_env_ref_in_config": false
   docker_config_update_interval	int	The minimum interval at which the container path is updated. Default value for versions earlier than Logtail V1.0.32: 10. Default value for Logtail V1.0.32 or later: 3. Unit: seconds. This parameter is used together with the max_docker_config_update_times parameter. If one of the values for the two parameters is reached, the container path is no longer updated.	"docker_config_update_interval": 3
   max_docker_config_update_times	int	The maximum number of times that the container path can be updated within 3 minutes. Default value for versions earlier than Logtail V1.0.32: 3. Default value for Logtail V1.0.32 or later: 10. By default, if the container path is updated more than three times within a 3-minute period, the container path cannot be updated again until 3 minutes later.	"max_docker_config_update_times": 10
   DOCKER_HOST	String	The socket address that is used to communicate with Docker. You must configure the socket address by using environment variables. This parameter is empty by default, which indicates that the default socket address unix:///var/run/docker.sock is used.	DOCKER_HOST=unix:///var/run/docker.sock
   CONTAINERD_SOCK_PATH	String	The socket address that is used to communicate with containerd. You must configure the socket address by using environment variables. This parameter is empty by default, which indicates that the default socket address unix:///run/containerd/containerd.sock is used. If a K3s cluster is used, you can change the default socket address to the value provided in the example.	CONTAINERD_SOCK_PATH=/run/k3s/containerd/containerd.sock
   logreader_max_rotate_queue_size	Int	The maximum length of the queue in which a file is rotated. Default value: 20. If log collection is blocked or delayed, the files from which you want to collect logs are assigned the file handles and wait in the queue. If log collection is delayed and you need to manage the maximum disk usage, you can set this parameter to a smaller value. Warning If the number of delayed files exceeds the value of this parameter, Logtail does not collect logs from new files.	"logreader_max_rotate_queue_size" : 10
   force_release_deleted_file_fd_timeout	Int	The timeout period for the release of a file handle. If you want to release a file handle after a container exits or a file is deleted, you can configure this parameter. Unit: seconds. Default value: -1, which indicates that the feature is disabled. The value 0 indicates that file handles are immediately released. If you want to manage the maximum destruction latency of containerd containers, you can configure this parameter. Warning If log collection is delayed and the latency exceeds the specified threshold, data that is not collected is lost.	"force_release_deleted_file_fd_timeout" : 0
   default_max_inotify_watch_num	Int	The maximum number of directories that are monitored by using inotify. The directories include subdirectories. Default value: 3000.	"default_max_inotify_watch_num" : 5000

3. Restart Logtail for the new settings to take effect.

   /etc/init.d/ilogtaild stop && /etc/init.d/ilogtaild start                        

   After you restart Logtail, you can run the /etc/init.d/ilogtaild status command to check the status of Logtail.

Appendix: Environment variables

The following table describes the mappings between environment variables and the startup parameters of Logtail. For information about the startup parameters of Logtail, see Startup parameters of Logtail.