Simple Log Service:Logtail network types, startup parameters, and configuration files

Network types and selection

When you install Logtail, you can select the appropriate network type based on the location of your server relative to Simple Log Service. This ensures faster and more stable log data transmission. The following examples show how to select a network type.

1. Log on to the Simple Log Service console. On the Project Overview page of your project, enable Acceleration Endpoint.

2. Read the prompt in the dialog box and click Confirm.

3. You can configure Logtail collection acceleration in one of the following ways.

   Install Logtail using transfer acceleration

   If you enable transfer acceleration before you install Logtail, you can install Logtail using the transfer acceleration feature. For more information, see Install Logtail.

   Modify the Logtail configuration

   If you installed Logtail before you enabled transfer acceleration, you can modify the Logtail configuration.

   1. Stop Logtail.

      • Linux

        Run the sudo /etc/init.d/ilogtaild stop command.

      • Windows

        1. Go to Start > Control Panel > Administrative Tools > Services.

        2. In the Services dialog box, find the LogtailDaemon service for Logtail 1.0.0.0 or later, or the LogtailWorker service for Logtail 0.x.x.x. Right-click the service and select Stop.

   2. Modify the Logtail startup configuration file ilogtail_config.json.

      In the data_server_list parameter, replace the endpoint line with log-global.aliyuncs.com. For more information about the file path, see Startup parameter configuration file (ilogtail_config.json).

   3. Start Logtail.

      • Linux

        Run the sudo /etc/init.d/ilogtaild start command.

      • Windows

        1. Go to Start > Control Panel > Administrative Tools > Services.

        2. In the Services dialog box, find the LogtailDaemon service for Logtail 1.0.0.0 or later, or the LogtailWorker service for Logtail 0.x.x.x. Right-click the service and select Start.

For more information about related operations, see Manage transfer acceleration.

Scenarios

Recommended parameter values

The following parameter settings are recommended based on practical experience and are suitable for collecting common JSON files. The performance of full regex mode and delimiter mode is similar to that of JSON mode. The performance of simple mode is five times that of JSON mode. The complexity of data and rules, and the number of collection directories and files affect CPU and memory consumption. You can adjust the parameters as needed based on the following tables and your actual situation.

• Host environment

  Parameter	Default collection rate	Collection rate greater than 10 MB/s	Collection rate greater than 20 MB/s	Collection rate greater than 40 MB/s
  cpu_usage_limit	0.4	1	2	4
  mem_usage_limit	384	1024	2048	4096
  max_bytes_per_sec	20971520	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	4	20	40	80

• Container or Kubernetes environment

  Environment variable	Default collection rate	Collection rate greater than 10 MB/s	Collection rate greater than 20 MB/s	Collection rate greater than 40 MB/s
  cpu_usage_limit	2	3	5	9
  mem_usage_limit	2048	2048	2048	4096
  max_bytes_per_sec	209715200	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	20	20	40	80
  resources.limits.cpu	500M	1000M	2000M	4000M
  resources.limits.memory	2 Gi	2 Gi	3 Gi	5 Gi

  The following section describes how to modify the startup parameters of Logtail in a container or Kubernetes environment.

  • If Logtail is deployed in an Alibaba Cloud Kubernetes (ACK) cluster and the Logtail version is Logtail-ds 1.7.3 or later, you can modify the parameters in the Container Service for Kubernetes (ACK) console. On the Add-ons page, modify the parameters in the logtail-ds component.

  • If Logtail is deployed in a self-managed container or Kubernetes environment and the Logtail version is Logtail-ds 1.7.3 or later, you can modify the Logtail startup parameters by modifying the daemonset environment variables. Some environments reference a ConfigMap. The path of the ConfigMap is configmap > kube-system > alibaba-log-configuration.

  • If the Logtail version is earlier than Logtail-ds 1.7.3, you can modify the Logtail startup parameters by modifying the daemonset environment variables. Some environments reference a ConfigMap. The path of the ConfigMap is configmap > kube-system > alibaba-log-configuration. You also need to adjust the resources.limits.cpu and resources.limits.memory parameters in daemonset > kube-system > logtail-ds to prevent the container from exceeding its resource limits.

When you configure Logtail startup parameters based on the values in the Collection rate greater than 40 MB/s column, the collection performance of Logtail approaches its limit. Adding more threads does not significantly improve performance. The following table describes the performance limits of the collection client.

Set startup parameters

1. On the server where Logtail is installed, open the /usr/local/ilogtail/ilogtail_config.json file.

   This step applies to host environments.

   In a container or Kubernetes environment, you can modify the Logtail startup parameters by modifying the daemonset environment variables. Some environments reference a ConfigMap. The path of the ConfigMap is configmap > kube-system > alibaba-log-configuration.

2. Set the startup parameters as needed.

   The following code shows an example of startup parameters.

   {
       ...
       "cpu_usage_limit" : 0.4,
       "mem_usage_limit" : 384,
       "max_bytes_per_sec" : 20971520,
       "process_thread_count" : 1,
       "send_request_concurrency" : 4,
       "buffer_file_num" : 25,
       "buffer_file_size" : 20971520,
       "buffer_file_path" : "",
       ...
   }

   Note

   • The following table describes only common startup parameters. For other parameters, you can keep the default configurations.

   • You can add or modify the specified startup parameters as needed.

   Table 1. Logtail startup parameters

   Parameter	Type	Description	Example
   cpu_usage_limit	double	The CPU utilization threshold, calculated on a single-core basis. Valid values are described as follows: Range: 0.1 to the number of CPU cores on the current machine Default value: 0.4 Warning The cpu_usage_limit parameter is a soft limit. The actual CPU utilization of Logtail may exceed the limit. If the limit is exceeded for 5 minutes, a circuit-breaking protection mechanism is triggered, and Logtail automatically restarts. For example, if you set this parameter to 0.4, Simple Log Service attempts to limit Logtail's CPU utilization to 40% of a single CPU core. If the utilization exceeds this limit, Logtail automatically restarts. Typically, a single core can process about 100 MB/s of data in simple mode and about 20 MB/s in full regex mode. Note If Logtail is deployed in an ACK cluster and the Logtail version is Logtail-ds 1.7.3 or later, set the CPU utilization threshold in the ACK console. On the Add-ons page, modify the LogtailDSLimitCPU parameter of the logtail-ds component. For more information, see Manage components.	"cpu_usage_limit" : 0.4
   mem_usage_limit	int	The memory usage threshold. Valid values are described as follows: Range: 128 MB to 8192 MB Default value: 384 MB (host), 2048 MB (ACK component) Warning The mem_usage_limit parameter is a soft limit. The actual memory usage of Logtail may exceed the limit. If the limit is exceeded for 5 minutes, a circuit-breaking protection mechanism is triggered, and Logtail automatically restarts. The collection rate, the number of monitored directories and files, and the degree of send blocking are related to the mem_usage_limit parameter. For more information, see Logtail limits. Note If Logtail is deployed in an ACK cluster and the Logtail version is Logtail-ds 1.7.3 or later, set the memory usage threshold in the ACK console. On the Add-ons page, modify the LogtailDSLimitMemory parameter of the logtail-ds component. For more information, see Manage components.	"mem_usage_limit" : 384
   max_bytes_per_sec	int	The traffic limit for sending raw data from Logtail per second. Valid values are described as follows: Range: 1024 Byte/s to 52428800 Byte/s Default value: 20971520 Byte/s Important If you set this parameter to a value greater than 20971520 Byte/s (20 MB/s), the speed is not limited. For example, if you set this parameter to 2097152, Logtail sends data at a rate of 2 MB/s. Note If Logtail is deployed in an ACK cluster and the Logtail version is Logtail-ds 1.7.3 or later, set the traffic limit in the ACK console. On the Add-ons page, modify the LogtailDSMaxBytePerSec parameter of the logtail-ds component. For more information, see Manage components.	"max_bytes_per_sec" : 2097152
   process_thread_count	int	The number of threads Logtail uses to process data. Valid values are described as follows: Range: 1 to 64 Default value: 1 Typically, a single thread can handle data writes of 24 MB/s in simple mode or 12 MB/s in full regex mode. You do not need to adjust this parameter value in most cases. Note If Logtail is deployed in an ACK cluster and the Logtail version is Logtail-ds 1.7.3 or later, set the number of threads in the ACK console. On the Add-ons page, modify the LogtailDSProcessThreadCount parameter of the logtail-ds component. For more information, see Manage components.	"process_thread_count" : 1
   send_request_concurrency	int	The number of asynchronous concurrent requests. Valid values are described as follows: Range: 1 to 50 Default value: 20 If the write TPS is high, you can set a higher number of asynchronous concurrent requests. You can calculate this based on one concurrent request supporting 0.5 MB/s to 1 MB/s of network throughput. The actual value depends on network latency. Note Setting the number of asynchronous concurrent requests too high can lead to excessive network port usage. Adjust TCP-related parameters. If Logtail is deployed in an ACK cluster and the Logtail version is Logtail-ds 1.7.3 or later, set the number of asynchronous concurrent requests in the ACK console. On the Add-ons page, modify the LogtailDSSendRequestConcurrency parameter of the logtail-ds component. For more information, see Manage components.	"send_request_concurrency" : 4
   buffer_file_num	int	The maximum number of cache files. Valid values are described as follows: Range: 1 to 100 Default value: 25 If network exceptions occur or write quotas are exceeded, Logtail writes the real-time parsed logs to local files in the installation directory for caching. It retries sending them after the issue is resolved.	"buffer_file_num" : 25
   buffer_file_size	int	The maximum size of a single cache file in bytes. Valid values are described as follows: Range: 1048576 Byte to 104857600 Byte Default value: 20971520 Byte The maximum disk space that can be used by cache files is buffer_file_size × buffer_file_num.	"buffer_file_size" : 20971520
   buffer_file_path	String	The directory where cache files are stored. The default value is empty, which means cache files are stored in the Logtail installation directory /usr/local/ilogtail. After you set this parameter, manually move the files named logtail\_buffer\_file_* from the original directory to this directory. This ensures that Logtail can read the cache files and delete them after they are sent.	"buffer_file_path" : ""
   bind_interface	String	The name of the network interface card (NIC) bound to the local machine. The default value is empty, which means an available NIC is automatically bound. If you set this to a specific NIC, such as eth1, Logtail is forced to use that NIC to upload logs. This parameter is supported only for Linux.	"bind_interface" : ""
   check_point_filename	String	The path where Logtail's checkpoint files are saved. Default value: /tmp/logtail_check_point. If you use Docker or Kubernetes, configure this parameter by referring to Data reliability of iLogtail after a container is restarted. This prevents data duplication or loss caused by checkpoint information loss when the Logtail container restarts.	"check_point_filename" : /tmp/logtail_check_point
   check_point_dump_interval	int	The interval at which Logtail updates checkpoint files. Default value: 900. Unit: seconds. By default, Logtail updates checkpoint files every 15 minutes. This parameter is supported only in Linux Logtail 1.0.19 and later or Windows Logtail 1.0.19.0 and later.	"check_point_dump_interval" : 900
   user_config_file_path	String	The path where Logtail configuration files are saved. By default, this is the directory of the process binary, and the file name is user_log_config.json. If you use Docker or Kubernetes, configure this parameter by referring to Data reliability of iLogtail after a container is restarted. This prevents data duplication or loss when the Logtail container restarts.	"user_config_file_path" : user_log_config.json
   docker_file_cache_path	String	This file records the path mappings from container files to host files. The default path is /usr/local/ilogtail/docker_path_config.json. If you use Docker or Kubernetes, configure this parameter by referring to Data reliability of iLogtail after a container is restarted. This prevents data duplication or loss when the Logtail container restarts. This parameter is supported only in Linux Logtail 0.16.54 and later or Windows Logtail 0.16.54.0 and later.	"docker_file_cache_path": /usr/local/ilogtail/docker_path_config.json
   discard_old_data	Boolean	Specifies whether to discard historical logs. Default value: true. This means logs older than 12 hours from the current time are discarded.	"discard_old_data" : true
   ilogtail_discard_interval	int	The threshold for discarding historical logs relative to the current time. Default value: 43200 (12 hours). Unit: seconds.	"ilogtail_discard_interval": 43200
   working_ip	String	The default value is empty, which means the IP address is automatically obtained from the local server. After modification, Logtail reports this value as the server's IP address.	"working_ip" : ""
   working_hostname	String	The hostname of the local server reported by Logtail. The default value is empty, which means the hostname is automatically obtained from the local server.	"working_hostname" : ""
   max_read_buffer_size	long	The maximum size of a single log entry that can be read. Default value: 524288 (512 KB). Maximum value: 8388608 (8 MB). Unit: Byte. If a single log entry exceeds 524288 Bytes, modify this parameter.	"max_read_buffer_size" : 524288
   oas_connect_timeout	long	The connection timeout for requests initiated by Logtail to get configurations, AccessKey pairs, and other information. Default value: 5. Unit: seconds. Modify this parameter if the connection takes too long to establish due to poor network conditions.	"oas_connect_timeout" : 5
   oas_request_timeout	long	The total request timeout for requests initiated by Logtail to get configurations, AccessKey pairs, and other information. Default value: 10. Unit: seconds. Modify this parameter if the connection takes too long to establish due to poor network conditions.	"" : 10
   data_server_port	long	After you set data_server_port to 443, Logtail transmits data to Simple Log Service over HTTPS. This parameter is supported only in Linux Logtail 1.0.10 and later or Windows Logtail 1.0.10.0 and later.	"data_server_port": 443
   enable_log_time_auto_adjust	Boolean	After you set enable_log_time_auto_adjust to true, the log time adapts to the server's local time. For data security, Simple Log Service validates the time carried by requests, including those from Logtail. It rejects requests with a time difference of more than 15 minutes from the server time. Logtail requests carry the server's local time. If the server's local time is changed, for example, to a future time for testing, Logtail requests are rejected, causing data write failures. Use this parameter to make the log time adapt to the server's local time. This parameter is supported only in Linux Logtail 1.0.19 and later or Windows Logtail 1.0.19.0 and later. Important When this feature is enabled, the offset between the Simple Log Service server time and the local server time is added to the log time. Because the offset is updated only when a request is rejected by the Simple Log Service server, the log time queried on the server may not be the same as the actual write time of the log. Some Logtail logic depends on the system time incrementing. Restart Logtail after each machine time adjustment.	"enable_log_time_auto_adjust": true
   accept_multi_config	Boolean	Specifies whether to allow multiple Logtail configurations to collect logs from the same file. Default value: false. This means it is not allowed. By default, a file can be collected by only one Logtail configuration. Use this parameter to remove the restriction. The processing of each Logtail configuration is independent. Allowing multiple Logtail configurations to collect logs from the same file consumes multiple times the CPU and memory overhead. This parameter is supported only in Linux Logtail 0.16.26 and later or Windows Logtail 0.16.26.0 and later.	"accept_multi_config": true
   enable_checkpoint_sync_write	Boolean	Specifies whether to enable the sync write feature. Default value: false. This means it is not enabled. The sync write feature is mainly used with the ExactlyOnce write feature. After the ExactlyOnce write feature is enabled, Logtail records fine-grained checkpoint information at the file level on the local disk. However, for performance reasons, the sync write feature is not called by default when writing checkpoints. Therefore, if the machine restarts and buffer data is not written to the disk in time, checkpoints may be lost. In this case, set enable_checkpoint_sync_write to true to enable the sync write feature. For more information, see Logtail configurations (legacy). This parameter is supported only in Linux Logtail 1.0.20 and later or Windows Logtail 1.0.20.0 and later.	"enable_checkpoint_sync_write": false
   enable_env_ref_in_config	Boolean	Specifies whether to enable the environment variable replacement feature for collection configurations. Default value: false. After you enable this feature, you can use ${xxx} as a placeholder for the environment variable xxx in the Logtail collection configuration in the console. For example, if you set the collection path to /${xxx}/logs and the environment variable to xxx=user, the effective collection path is /user/logs. To use ${ or } in the configuration, use $${ or $} to escape them. This parameter is supported only in Linux Logtail 1.0.31 and later or Windows Logtail 1.0.31.0 and later.	"enable_env_ref_in_config": false
   docker_config_update_interval	int	The minimum time interval for updating container paths. This parameter is used with max_docker_config_update_times. If the threshold of either parameter is reached, the container path is no longer updated. For Linux Logtail 1.0.32 and later or Windows Logtail 1.0.32.0 and later, the default value is 3. Unit: seconds. For Linux Logtail earlier than 1.0.32 or Windows Logtail earlier than 1.0.32.0, the default value is 10. Unit: seconds.	"docker_config_update_interval": 3
   max_docker_config_update_times	int	The maximum number of times a container path can be updated within 3 minutes. By default, if the container path is updated more than 3 times in 3 minutes, it will not be updated again. For Linux Logtail 1.0.32 and later or Windows Logtail 1.0.32.0 and later, the default value is 10. For Linux Logtail earlier than 1.0.32 or Windows Logtail earlier than 1.0.32.0, the default value is 3.	"max_docker_config_update_times": 10
   DOCKER_HOST	String	The socket address for communicating with Docker. Configure this using an environment variable. The default value is empty, which means the default address unix:///var/run/docker.sock is used.	DOCKER_HOST=unix:///var/run/docker.sock
   CONTAINERD_SOCK_PATH	String	The socket address for communicating with containerd. Configure this using an environment variable. The default value is empty, which means the default address unix:///run/containerd/containerd.sock is used. If it is a K3s cluster, you can modify it as shown in the example.	CONTAINERD_SOCK_PATH=/run/k3s/containerd/containerd.sock
   logreader_max_rotate_queue_size	Int	The maximum length of the rotation queue. Default value: 20. When log collection is blocked or delayed, the files to be collected hold file handles and wait in the queue. When collection is delayed, consider reducing this value to control the maximum disk usage. Warning When the number of delayed files exceeds this value, Logtail skips the collection of new files.	"logreader_max_rotate_queue_size" : 10
   force_release_deleted_file_fd_timeout	Int	The handle is released within a certain period after a container exits or a file is deleted. You can specify the time. Default value: -1, which means the feature is disabled. A value of 0 means immediate release. Unit: seconds. If you want to control the maximum destruction delay of containerd containers, consider setting this parameter. Warning When collection is delayed, data delayed beyond the configured time is lost.	"force_release_deleted_file_fd_timeout" : 0
   data_endpoint_policy	string	The switchover policy for Logtail to access Simple Log Service endpoints. The following values are optional: Note In the ilogtail_config.json file, check the data_server_list parameter to see if a default domain name is configured. For more information, see Startup parameter configuration file (ilogtail_config.json). designated_first (default) If a default domain name is specified for a region and is available, the system uses the default domain name first. If a default domain name is specified for a region but is unavailable, the system automatically selects an available domain name. If no default domain name is specified for a region, the system automatically selects an available domain name. designated_locked If a default domain name is specified for a region, the system uses only the default domain name, regardless of its availability. If no default domain name is specified for a region, the system automatically selects an available domain name. This parameter is supported only in Linux Logtail 1.5.0 and later or Windows Logtail 1.5.0.0 and later.	"data_endpoint_policy" : "designated_first"
   inotify_black_list	Array<String>	The inotify listener blacklist. The blacklist uses exact matching. Directories in this list do not enable inotify listening.	"inotify_black_list": ["/tmp"]
   host_path_blacklist	String	The global host path blacklist. The blacklist uses substring matching. In Linux, separate multiple substrings with a colon (:). In Windows, separate multiple substrings with a semicolon (;). For example, "host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-" prohibits the collection of NAS mount data. This parameter is supported only in Linux Logtail 1.8.0 and later or Windows Logtail 1.8.0.0 and later.	"host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-"
   LOGTAIL_LOG_LEVEL	String	The log printing level. Configure this using an environment variable. The default value is empty, which means info. Optional values are trace, debug, info, warning, error, and fatal. This parameter is supported only in Linux Logtail 1.8.0 and later or Windows Logtail 1.8.0.0 and later.	LOGTAIL_LOG_LEVEL=info
   FORCE_RELEASE_STOP_CONTAINER_FILE	Boolean	Configuration method: This parameter can be configured only using an environment variable. Feature description: When this parameter is set to true, Logtail immediately releases container file handles when the application container exits. This operation prevents the container from failing to exit normally because file handles are not released. Notes: In this case, the completeness of data collection from the container cannot be guaranteed. Add a delay of a few seconds before the application exits to ensure that logs can be completely collected. Supported versions: Linux Logtail 2.1.6 and later	"FORCE_RELEASE_STOP_CONTAINER_FILE" : "true"
   default_reader_flush_timeout	int	The timeout period for determining the completeness of the last log line. Default value: 60. Unit: seconds. This parameter is supported only in Logtail 2.0.0 and later.	"default_reader_flush_timeout" : 1

3. Restart Logtail for the configuration to take effect.

   /etc/init.d/ilogtaild stop && /etc/init.d/ilogtaild start                        

   After the restart, you can run the /etc/init.d/ilogtaild status command to check the status of Logtail.

Appendix: Environment variables

The following table describes the mapping between environment variables and Logtail startup parameters.

Startup parameter configuration file (ilogtail_config.json)

• File path

  • Host environment

    Operating system	Logtail	ilogtail_config.json file path
    Linux	Logtail (64-bit program)	/usr/local/ilogtail/ilogtail_config.json
    Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json
    	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json Note A 64-bit Windows OS supports both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications.
    Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

  • Container environment

    The ilogtail_config.json file is stored in the Logtail container. The file path is configured in the ALIYUN_LOGTAIL_CONFIG environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_CONFIG command to view the path. For example: /etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json.

• File example

  $cat /usr/local/ilogtail/ilogtail_config.json
  {
      "config_server_address" : "http://logtail.cn-hangzhou-intranet.log.aliyuncs.com",
      "data_server_list" :
      [
          {
              "cluster" : "cn-hangzhou",
              "endpoint" : "cn-hangzhou-intranet.log.aliyuncs.com"
          }
      ],
      "cpu_usage_limit" : 0.4,
      "mem_usage_limit" : 100,
      "max_bytes_per_sec" : 2097152,
      "process_thread_count" : 1,
      "send_request_concurrency" : 4,
      "streamlog_open" : false
  }

User identifier configuration file

The user identifier configuration file contains the ID of your Alibaba Cloud account. This ID indicates that the account has permission to access this server and collect its logs. For more information, see Configure a user identifier.

• File path

  • Host environment

    • Linux: /etc/ilogtail/users/

    • Windows: C:\LogtailData\users\

  • Container environment

    The user identifier is saved in the ALIYUN_LOGTAIL_USER_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_ID command to view the user identifier.

• File example

  $ls /etc/ilogtail/users/

Custom identifier file (user_defined_id)

The user_defined_id file is used to configure a custom ID. For more information, see Create a machine group based on a custom ID.

• File path

  • Host environment

    • Linux: /etc/ilogtail/user_defined_id

    • Windows: C:\LogtailData\user_defined_id

  • Container environment

    The custom ID is configured in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view the custom ID.

• File example

  $cat /etc/ilogtail/user_defined_id
  aliyun-ecs-rs1e16355

Logtail collection configuration file (user_log_config.json)

The user_log_config.json file records the Logtail collection configuration information that is obtained from Simple Log Service. The file is in JSON format and is updated synchronously whenever the Logtail collection configuration is updated. You can use the user_log_config.json file to confirm whether the Logtail collection configuration has been delivered to the server. If the Logtail collection configuration file exists and its content is consistent with the Logtail collection configuration on Simple Log Service, the configuration has been delivered.

• File path

  • Host environment

    Operating system	Logtail	user_log_config.json file path
    Linux	Logtail (64-bit program)	/usr/local/ilogtail/user_log_config.json
    Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\user_log_config.json
    	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\user_log_config.json Note A 64-bit Windows OS supports both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications.
    Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\user_log_config.json

  • Container environment

    The user_log_config.json file is stored in the Logtail container at the path /usr/local/ilogtail/user_log_config.json.

• File example

  $cat /usr/local/ilogtail/user_log_config.json
  {
     "metrics" : {
        "##1.0##k8s-log-c12ba2028*****939f0b$app-java" : {
           "aliuid" : "16542189*****50",
           "category" : "app-java",
           "create_time" : 1534739165,
           "defaultEndpoint" : "cn-hangzhou-intranet.log.aliyuncs.com",
           "delay_alarm_bytes" : 0,
           "enable" : true,
           "enable_tag" : true,
           "filter_keys" : [],
           "filter_regs" : [],
           "group_topic" : "",
           "local_storage" : true,
           "log_type" : "plugin",
           "log_tz" : "",
           "max_send_rate" : -1,
           "merge_type" : "topic",
           "plugin" : {
              "inputs" : [
                 {
                    "detail" : {
                       "IncludeEnv" : {
                          "aliyun_logs_app-java" : "stdout"
                       },
                       "IncludeLable" : {
                          "io.kubernetes.container.name" : "java-log-demo-2",
                          "io.kubernetes.pod.namespace" : "default"
                       },
                       "Stderr" : true,
                       "Stdout" : true
                    },
                    "type" : "service_docker_stdout"
                 }
              ]
           },
           "priority" : 0,
           "project_name" : "k8s-log-c12ba2028c*****ac1286939f0b",
           "raw_log" : false,
           "region" : "cn-hangzhou",
           "send_rate_expire" : 0,
           "sensitive_keys" : [],
           "tz_adjust" : false,
           "version" : 1
        }
     }
  }

AppInfo record file (app_info.json)

The app_info.json file records the startup time of Logtail, the obtained IP address, the hostname, and other information.

• File path

  • Host environment

    Operating system	Logtail	app_info.json file path
    Linux	Logtail (64-bit program)	/usr/local/ilogtail/app_info.json
    Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\app_info.json
    	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json Note A 64-bit Windows OS supports both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications.
    Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\app_info.json

  • Container environment

    The app_info.json file is stored in the Logtail container at the path /usr/local/ilogtail/app_info.json.

• File example

  $cat /usr/local/ilogtail/app_info.json
  {
     "UUID" : "",
     "hostname" : "logtail-ds-slpn8",
     "instance_id" : "E5F93BC6-B024-11E8-8831-0A58AC14039E_1**.***.***.***_1536053315",
     "ip" : "1**.***.***.***",
     "logtail_version" : "0.16.13",
     "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
     "update_time" : "2018-09-04 09:28:36"
  }

  Field	Description
  UUID	The server serial number.
  hostname	The hostname.
  instance_id	The randomly generated unique identifier for Logtail.
  ip	The IP address obtained by Logtail. If this field is empty, Logtail has not obtained an IP address and cannot run normally. Set an IP address for the server and restart Logtail. Note If you create an IP address-based machine group, make sure that the IP address configured in the machine group is the same as the IP address displayed here. If they are not the same, confirm the correct host IP address. Then, either modify the IP address in the machine group in the Simple Log Service console or modify the working_ip parameter in Set Logtail startup parameters.
  logtail_version	The Logtail client version.
  os	The operating system version.
  update_time	The last time Logtail was started.

Logtail operational log (ilogtail.LOG)

The ilogtail.LOG file records the operational logs of Logtail. The log levels, from lowest to highest, are INFO, WARN, and ERROR. You can ignore logs of the INFO level.

• File path

  • Host environment

    Operating system	Logtail	ilogtail.LOG file path
    Linux	Logtail (64-bit program)	/usr/local/ilogtail/ilogtail.LOG
    Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG
    	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail.LOG Note A 64-bit Windows OS supports both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications.
    Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG

  • Container environment:

    The ilogtail.LOG file is stored in the Logtail container at the path /usr/local/ilogtail/ilogtail.LOG.

• File example

  $tail /usr/local/ilogtail/ilogtail.LOG
  [2018-09-13 01:13:59.024679]    [INFO]    [3155]    [build/release64/sls/ilogtail/elogtail.cpp:123]    change working dir:/usr/local/ilogtail/
  [2018-09-13 01:13:59.025443]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:175]    load logtail config file, path:/etc/ilogtail/conf/ap-southeast-1/ilogtail_config.json
  [2018-09-13 01:13:59.025460]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:176]    load logtail config file, detail:{
     "config_server_address" : "http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
     "data_server_list" : [
        {
           "cluster" : "ap-southeast-1",
           "endpoint" : "ap-southeast-1-intranet.log.aliyuncs.com"
        }
  ]

Logtail plugin log (logtail_plugin.LOG)

The logtail_plugin.LOG file records the operational logs of Logtail plugins. The log levels, from lowest to highest, are INFO, WARN, and ERROR. You can ignore logs of the INFO level.

If an error such as CANAL_RUNTIME_ALARM occurs during error diagnosis, you can use the logtail_plugin.LOG file to troubleshoot the error.

• File path

  • Host environment

    Operating system	Logtail	logtail_plugin.LOG file path
    Linux	Logtail (64-bit program)	/usr/local/ilogtail/logtail_plugin.LOG
    Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG
    	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\logtail_plugin.LOG Note A 64-bit Windows OS supports both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications.
    Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG

  • Container environment

    The logtail_plugin.LOG file is stored in the Logtail container at the path /usr/local/ilogtail/logtail_plugin.LOG.

• File example

  $tail /usr/local/ilogtail/logtail_plugin.LOG
  2018-09-13 02:55:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 02:55:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
  2018-09-13 03:00:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 03:00:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
  2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
  2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
  2018-09-13 03:04:26 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
  2018-09-13 03:04:27 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
  2018-09-13 03:05:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 03:05:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop

Container path mapping file (docker_path_config.json)

The docker_path_config.json file is created only when collecting container logs. It records the path mapping relationship between container files and host files. The file is in JSON format.

If a DOCKER_FILE_MAPPING_ALARM error occurs during error diagnosis, it indicates a failure to add a Docker file mapping. You can use the docker_path_config.json file to troubleshoot the error.

• File path

  /usr/local/ilogtail/docker_path_config.json

• File example

  $cat /usr/local/ilogtail/docker_path_config.json
  {
     "detail" : [
        {
           "config_name" : "##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$nginx",
           "container_id" : "df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10",
           "params" : "{\n   \"ID\" : \"df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10\",\n   \"Path\" : \"/logtail_host/var/lib/docker/overlay2/947db346695a1f65e63e582ecfd10ae1f57019a1b99260b6c83d00fcd1892874/diff/var/log\",\n   \"Tags\" : [\n      \"nginx-type\",\n      \"access-log\",\n      \"_image_name_\",\n      \"registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest\",\n      \"_container_name_\",\n      \"nginx-log-demo\",\n      \"_pod_name_\",\n      \"nginx-log-demo-h2lzc\",\n      \"_namespace_\",\n      \"default\",\n      \"_pod_uid_\",\n      \"87e56ac3-b65b-11e8-b172-00163f008685\",\n      \"_container_ip_\",\n      \"172.20.4.224\",\n      \"purpose\",\n      \"test\"\n   ]\n}\n"
        } 
     ],
     "version" : "0.1.0"
  }