Simple Log Service:Logtail parameters and configuration files

Network types and selection

When installing Logtail, select a network type based on the server's location relative to your SLS project for faster, more stable data transfer.

1. Log on to the Log Service console. Go to the Project Overview page and enable Acceleration Endpoint as shown in the figure below.

2. Read the prompt in the dialog box and click Modify.

3. Configure transfer acceleration for Logtail in one of the following ways.

   Install Logtail with transfer acceleration

   If transfer acceleration is already enabled, select the transfer acceleration option during Logtail installation. Install Logtail.

   Modify Logtail

   If you enable transfer acceleration after Logtail is installed, modify the Logtail configuration.

   1. Stop Logtail.

      • Linux

        Run the sudo /etc/init.d/ilogtaild stop command.

      • Windows

        1. Go to Start > Control Panel > Tools > Service.

        2. In the Service dialog box, find the LogtailDaemon service (for Logtail 1.0.0.0 or later) or the LogtailWorker service (for Logtail 0.x.x.x versions), right-click the service, and then select Stop.

   2. Modify the startup parameter configuration file, ilogtail_config.json.

      In the data_server_list parameter, replace the existing endpoint value with log-global.aliyuncs.com. For the location of this file, see startup parameter configuration file (ilogtail_config.json).

   3. Start Logtail.

      • Linux

        Run the sudo /etc/init.d/ilogtaild start command.

      • Windows

        1. Go to Start > Control Panel > Tools > Service.

        2. In the Service dialog box, find the LogtailDaemon service (for Logtail 1.0.0.0 or later) or the LogtailWorker service (for Logtail 0.x.x.x versions), right-click the service, and then select Start.

Manage Transfer Acceleration.

Use cases

Recommended parameters

The following parameters are recommended for collecting logs from typical JSON files. The performance of full regex mode and delimiter mode is similar to that of JSON mode, while simple mode is approximately five times faster. Data complexity, parsing rules, and the number of monitored directories and files can affect CPU and memory consumption. Adjust the parameters based on your actual workload.

• Server environment

  Parameter	Default rate	Rate > 10 MB/s	Rate > 20 MB/s	Rate > 40 MB/s
  cpu_usage_limit	0.4	1	2	4
  mem_usage_limit	384	1024	2048	4096
  max_bytes_per_sec	20971520	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	4	20	40	80

• Container or Kubernetes environment

  Environment variable	Default rate	Rate > 10 MB/s	Rate > 20 MB/s	Rate > 40 MB/s
  cpu_usage_limit	2	3	5	9
  mem_usage_limit	2048	2048	2048	4096
  max_bytes_per_sec	209715200	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	20	20	40	80
  resources.limits.cpu	500m	1000m	2000m	4000m
  resources.limits.memory	2 Gi	2 Gi	3 Gi	5 Gi

  To modify startup parameters in a container or Kubernetes environment:

  • If Logtail-ds 1.7.3 or later is deployed in an Alibaba Cloud Kubernetes cluster, modify the parameters in the Container Service for Kubernetes console. On the Add-ons page, modify the parameters for the logtail-ds add-on.

  • If Logtail is deployed in a self-managed container or Kubernetes environment and the Logtail-ds version is 1.7.3 or later, you must modify the Logtail startup parameters by modifying the daemonset environment variables. Some environments reference a configmap, and the path to the configmap is configmap > kube-system > alibaba-log-configuration.

  • If you are using a version of Logtail-ds earlier than 1.7.3, you need to modify the Logtail startup parameters by editing the daemonset environment variables. Some environments reference a configmap, and the path to the configmap is configmap > kube-system > alibaba-log-configuration. You also need to adjust the resources.limits.cpu and resources.limits.memory parameters in the daemonset > kube-system > logtail-ds to avoid exceeding container resource limits.

Using the parameters from the Collection rate greater than 40 MB/s column, Logtail approaches its performance ceiling. The following table lists the upper limit for each collection mode.

Startup parameters

1. On the server where Logtail is installed, open the /usr/local/ilogtail/ilogtail_config.json file.

   This step applies only to host environments.

   In container or Kubernetes environments, you must modify the Logtail startup parameters by editing the DaemonSet environment variables. Some environments reference a ConfigMap, such as configmap > kube-system > alibaba-log-configuration.

2. Configure the startup parameters as needed.

   The following example shows the startup parameters:

   Note

   • The table lists common startup parameters only. Keep defaults for unlisted parameters.

   • You can add or modify startup parameters as needed.

   Table 1. Logtail startup parameters

   Parameter	Type	Description	Example
   cpu_usage_limit	double	The CPU utilization threshold for Logtail, relative to a single CPU core. Valid values: 0.1 to the number of CPU cores on the server. Default value: 0.4 Warning cpu_usage_limit is a soft limit. The actual CPU usage of Logtail may exceed this limit. If the limit is exceeded for 5 minutes, circuit breaking is triggered and Logtail automatically restarts. For example, if you set the value to 0.4, Log Service attempts to limit the CPU usage of Logtail to 40% of a single CPU core. If the CPU usage exceeds this limit, Logtail automatically restarts. Typically, a single core can process data at a rate of about 100 MB/s in simple mode or 20 MB/s in full regex mode. Note If Logtail-ds 1.7.3 or later is deployed in a Container Service for Kubernetes (ACK) cluster, we recommend that you set the CPU utilization threshold on the Container Service for Kubernetes console. On the Add-ons page, modify the LogtailDSLimitCPU parameter for the logtail-ds component. Manage components.	"cpu_usage_limit" : 0.4
   mem_usage_limit	int	The memory usage threshold for Logtail. Valid values: 128 MB to 8192 MB Default value: 384 MB (host) or 2048 MB (ACK components) Warning mem_usage_limit is a soft limit. The actual memory used by Logtail may exceed the limit. If the limit is exceeded for 5 minutes, circuit breaking is triggered, and Logtail restarts automatically. The collection rate, monitored directory and file count, and send blockage depend on mem_usage_limit. Logtail limits. Note If Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you set the memory usage threshold on the Container Service for Kubernetes console. On the Add-ons page, modify the LogtailDSLimitMemory parameter for the logtail-ds component. Manage components.	"mem_usage_limit" : 384
   max_bytes_per_sec	int	The maximum rate (in bytes per second) at which Logtail sends raw data. Valid values: 1,024 bytes/s to 52,428,800 bytes/s Default value: 20,971,520 bytes/s Important A value greater than 20,971,520 bytes/s (20 MB/s) means the rate is unlimited. For example, if you set the value to 2097152, Logtail sends data at a rate of 2 MB/s. Note If Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you set the flow limit on the Container Service for Kubernetes console. On the Add-ons page, modify the LogtailDSMaxBytePerSec parameter for the logtail-ds component. Manage components.	"max_bytes_per_sec" : 2097152
   process_thread_count	int	The number of threads that Logtail uses to process data. Valid values: 1 to 64 Default value: 1 Typically, a single thread can handle a write rate of 24 MB/s in simple mode or 12 MB/s in full regex mode. You typically do not need to change this parameter. Note If Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you set the number of threads on the Container Service for Kubernetes console. On the Add-ons page, modify the LogtailDSProcessThreadCount parameter for the logtail-ds component. Manage components.	"process_thread_count" : 1
   send_request_concurrency	int	The number of concurrent requests for asynchronous data sending. Valid values: 1 to 50 Default value: 20 If your write TPS is high, you can increase this value. A single concurrent request supports a network throughput of 0.5 MB/s to 1 MB/s, depending on the network latency. Note Setting this value too high may cause excessive network port usage. In this case, you may need to adjust TCP-related parameters. If Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you set the number of concurrent requests on the Container Service for Kubernetes console. On the Add-ons page, modify the LogtailDSSendRequestConcurrency parameter for the logtail-ds component. Manage components.	"send_request_concurrency" : 4
   buffer_file_num	int	The maximum number of buffer files that can be stored on disk. Valid values: 1 to 100 Default value: 25 If a network error occurs or the write quota is exceeded, Logtail caches parsed logs to local buffer files in the installation directory. After the issue is resolved, Logtail retries sending the data.	"buffer_file_num" : 25
   buffer_file_size	int	The maximum size of a single buffer file, in bytes. Valid values: 1,048,576 to 104,857,600 Default value: 20,971,520 buffer_file_size*buffer_file_num is the maximum disk space available for buffer files.	"buffer_file_size" : 20971520
   buffer_file_path	String	The directory where buffer files are stored. The default value is empty, which means that buffer files are stored in the Logtail installation directory /usr/local/ilogtail. After you set this parameter, you must manually move the file named logtail\_buffer\_file_* from the original directory to this directory to ensure that Logtail can read the buffer file and delete it after sending.	"buffer_file_path" : ""
   bind_interface	String	The name of the network interface controller (NIC) to which Logtail is bound. By default, this parameter is empty, and Logtail automatically binds to an available NIC. If you specify a NIC such as eth1, Logtail uses only that NIC to upload logs. This parameter is supported only on Linux.	"bind_interface" : ""
   check_point_filename	String	The path to the Logtail checkpoint file. The default value is /tmp/logtail_check_point. If you use Docker or Kubernetes, we recommend that you follow the instructions in Ensure data reliability when a Logtail container restarts to configure this parameter. This prevents data loss or duplication due to the loss of checkpoint information when the Logtail container restarts.	"check_point_filename" : "/tmp/logtail_check_point"
   check_point_dump_interval	int	The interval, in seconds, at which Logtail updates the checkpoint file. The default value is 900 (15 minutes). This parameter is supported only by Linux Logtail 1.0.19 or later and Windows Logtail 1.0.19.0 or later.	"check_point_dump_interval" : 900
   user_config_file_path	String	The save path for the Logtail configuration file. By default, the path is the directory where the process binary is located, and the filename is user_log_config.json. If you use Docker or Kubernetes, we recommend that you follow the instructions in Ensure data reliability when a Logtail container restarts to configure this parameter. This prevents data loss or duplication when the Logtail container restarts.	"user_config_file_path" : "user_log_config.json"
   docker_file_cache_path	String	This file records the path mappings from container files to host files, with the default path being /usr/local/ilogtail/docker_path_config.json. If you use Docker or Kubernetes, we recommend that you follow the instructions in Ensure data reliability when a Logtail container restarts to configure this parameter. This prevents data loss or duplication when the Logtail container restarts. This parameter is supported only by Linux Logtail 0.16.54 or later and Windows Logtail 0.16.54.0 or later.	"docker_file_cache_path": "/usr/local/ilogtail/docker_path_config.json"
   discard_old_data	Boolean	Specifies whether to discard historical logs. Default value: true. If this parameter is set to true, logs generated more than 12 hours ago are discarded.	"discard_old_data" : true
   ilogtail_discard_interval	int	The time threshold, in seconds, for discarding historical logs. If the difference between a log's timestamp and the current time exceeds this value, Logtail discards the log. The default value is 43200 (12 hours).	"ilogtail_discard_interval": 43200
   working_ip	String	The IP address that Logtail reports for the server. If this parameter is empty (the default), Logtail automatically obtains the IP address. Otherwise, it reports the specified value.	"working_ip" : ""
   working_hostname	String	The hostname that Logtail reports for the server. By default, this parameter is empty, and Logtail obtains the hostname from the server.	"working_hostname" : ""
   max_read_buffer_size	long	The maximum size of a single log entry that Logtail can read. Default value: 524,288 (512 KB). Maximum value: 8,388,608 (8 MB). Unit: bytes. If a single log entry exceeds 524,288 bytes, you can increase this value.	"max_read_buffer_size" : 524288
   oas_connect_timeout	long	The connection timeout for requests to obtain configurations or AccessKey pairs. Default value: 5. Unit: seconds. You can increase this value if connections take too long to establish due to poor network conditions.	"oas_connect_timeout" : 5
   oas_request_timeout	long	The total request timeout for requests that Logtail sends to obtain configurations or AccessKey pairs. Default value: 10. Unit: seconds. You can increase this value if connections take too long to establish due to poor network conditions.	"oas_request_timeout" : 10
   data_server_port	long	After you set data_server_port to 443, Logtail will transmit data to Log Service by using HTTPS. This parameter is supported only by Linux Logtail 1.0.10 or later and Windows Logtail 1.0.10.0 or later.	"data_server_port": 443
   enable_log_time_auto_adjust	Boolean	When enable_log_time_auto_adjust is set to true, the log time is automatically adjusted to the local time of the server. To ensure data security, Log Service validates the timestamp of each request and rejects those with a time skew of more than 15 minutes from the Log Service server time. Logtail uses the local time of the server for its requests. If the server's local time is modified (for example, to a future time for testing), Logtail requests may be rejected, causing data write failures. This parameter allows the log time to automatically adapt to the local time of the server to resolve such issues. This parameter is supported only by Linux Logtail 1.0.19 or later and Windows Logtail 1.0.19.0 or later. Important When this feature is enabled, the time offset between the Log Service server and the local server is added to the log time. This offset is updated only when a request is rejected by Log Service. As a result, the time of a log queried in Log Service may not match its actual write time. Some Logtail operations rely on monotonically increasing system time. We recommend restarting Logtail each time you adjust the machine's time.	"enable_log_time_auto_adjust": true
   accept_multi_config	Boolean	Specifies whether to allow multiple Logtail configurations to collect logs from the same file. Default value: false. By default, a file can be collected by only one Logtail configuration. This parameter removes this restriction. Each Logtail configuration is processed independently. Allowing multiple configurations to collect from the same file consumes proportionally more CPU and memory resources. This parameter is supported only by Linux Logtail 0.16.26 or later and Windows Logtail 0.16.26.0 or later.	"accept_multi_config": true
   enable_checkpoint_sync_write	Boolean	Specifies whether to enable synchronous checkpoint writing. Default value: false. The sync write feature is primarily used with the ExactlyOnce write feature. After you enable the ExactlyOnce write feature, Logtail records fine-grained checkpoint information at the file level on the local disk. However, for performance reasons, checkpoints are not immediately flushed to the disk by using a sync call by default. This may cause checkpoint loss if the machine restarts before the data in the buffer is written to the disk. In this case, you can set enable_checkpoint_sync_write to true to enable the sync write feature. Logtail Configuration (Legacy). This parameter is supported only by Linux Logtail 1.0.20 or later and Windows Logtail 1.0.20.0 or later.	"enable_checkpoint_sync_write": false
   enable_env_ref_in_config	Boolean	Specifies whether to enable environment variable replacement in collection configurations. Default value: false. After you enable this feature, you can use ${xxx} as a placeholder for the environment variable xxx in the Logtail collection configuration on the console. For example, if you set the collection path to /${xxx}/logs and the environment variable is xxx=user, the effective collection path is /user/logs. If you need to use ${ or } in your configuration, you can use $${ and $} to escape them. This parameter is supported only by Linux Logtail 1.0.31 or later and Windows Logtail 1.0.31.0 or later.	"enable_env_ref_in_config": false
   docker_config_update_interval	int	The minimum interval for updating container paths. When used with max_docker_config_update_times, the container path is no longer updated if either parameter reaches its threshold. For Linux Logtail 1.0.32 or later and Windows Logtail 1.0.32.0 or later, the default value is 3. Unit: seconds. For versions earlier than Linux Logtail 1.0.32 and Windows Logtail 1.0.32.0, the default value is 10. Unit: seconds.	"docker_config_update_interval": 3
   max_docker_config_update_times	int	The maximum number of times the container path can be updated within a 3-minute period. By default, if the container path is updated more than the specified number of times in 3 minutes, further updates are blocked. For Linux Logtail 1.0.32 or later and Windows Logtail 1.0.32.0 or later, the default value is 10. For versions earlier than Linux Logtail 1.0.32 and Windows Logtail 1.0.32.0, the default value is 3.	"max_docker_config_update_times": 10
   DOCKER_HOST	String	The socket address for communicating with Docker. This must be configured as an environment variable. By default, this parameter is empty, and the default address unix:///var/run/docker.sock is used.	DOCKER_HOST=unix:///var/run/docker.sock
   CONTAINERD_SOCK_PATH	String	The socket address for communicating with Containerd. This must be configured as an environment variable. By default, this parameter is empty, and the default address unix:///run/containerd/containerd.sock is used. For K3s clusters, modify the address as shown in the example.	CONTAINERD_SOCK_PATH=/run/k3s/containerd/containerd.sock
   logreader_max_rotate_queue_size	Int	The maximum length of the rotation queue. Default value: 20. When log collection is blocked or delayed, files pending collection are held in a queue with an open file handle. To control maximum disk usage during collection delays, consider reducing this value. Warning If the number of delayed files exceeds this value, Logtail skips collecting logs from new files.	"logreader_max_rotate_queue_size" : 10
   force_release_deleted_file_fd_timeout	Int	Specifies the timeout for releasing the file handle of a deleted file or a file in a stopped container. Default value: -1, which disables the feature. A value of 0 means the handle is released immediately. Unit: seconds. Set this parameter to control the maximum destruction latency for Containerd containers. Warning If collection is delayed, any data not collected before the timeout is lost.	"force_release_deleted_file_fd_timeout" : 0
   data_endpoint_policy	string	The policy for switching Log Service endpoints. Valid values: Note Check whether a default domain name is set in data_server_list in ilogtail_config.json. Startup parameter configuration file (ilogtail_config.json). designated_first (default) If a default endpoint is specified for a region and is available, it is prioritized. If the specified default endpoint is unavailable, the system automatically selects an available endpoint. If no default endpoint is specified, the system automatically selects an available endpoint. designated_locked If a default endpoint is specified for a region, the system uses only that endpoint, regardless of its availability. If no default endpoint is specified, the system automatically selects an available endpoint. This parameter is supported only by Linux Logtail 1.5.0 or later and Windows Logtail 1.5.0.0 or later.	"data_endpoint_policy" : "designated_first"
   inotify_black_list	Array<String>	An inotify monitoring blacklist. Logtail does not monitor directories that exactly match a path in this list.	"inotify_black_list": ["/tmp"]
   host_path_blacklist	String	A global blacklist for host paths, matched by substring. On Linux, use colons (:) to separate multiple substrings. On Windows, use semicolons (;) to separate multiple substrings. For example, "host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-" prevents the collection of data from NAS mounts. This parameter is supported only by Linux Logtail 1.8.0 or later and Windows Logtail 1.8.0.0 or later.	"host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-"
   LOGTAIL_LOG_LEVEL	String	The log level for Logtail's operational logs. This must be configured as an environment variable. By default, this parameter is empty, which corresponds to the info level. Valid values: trace, debug, info, warning, error, and fatal. This parameter is supported only by Linux Logtail 1.8.0 or later and Windows Logtail 1.8.0.0 or later.	LOGTAIL_LOG_LEVEL=info
   FORCE_RELEASE_STOP_CONTAINER_FILE	Boolean	Configuration method: This parameter can be configured only as an environment variable. When this parameter is set to true, Logtail immediately releases the container file handles when the application container exits. This prevents the container from failing to exit properly due to unreleased file handles. Notes: Data collection integrity from the container is not guaranteed. We recommend that you add a delay of a few seconds before the workload exits to ensure logs are fully collected. Supported versions: Linux Logtail 2.1.6 or later	"FORCE_RELEASE_STOP_CONTAINER_FILE" : "true"
   default_reader_flush_timeout	int	The timeout for determining if the last log entry is complete. Default value: 60. Unit: seconds. This parameter is supported only by Logtail 2.0.0 or later.	"default_reader_flush_timeout" : 1

3. Restart Logtail for the configuration to take effect.

   /etc/init.d/ilogtaild stop && /etc/init.d/ilogtaild start

   After the restart, run /etc/init.d/ilogtaild status to check the status of Logtail.

Appendix: Environment variables

This table lists the mappings between environment variables and Logtail startup parameters.

Startup parameter file (ilogtail_config.json)

• File path

  • Server environment

    Operating system	Logtail	Path
    Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail_config.json
    Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json
    	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json Note You can run both 32-bit and 64-bit applications on a 64-bit Windows operating system. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
    Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

  • Container environment

    The ilogtail_config.json file is stored in the Logtail container. The file path is configured in the ALIYUN_LOGTAIL_CONFIG environment variable. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_CONFIG command to view the file path. For example: /etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json.

• Example

  $cat /usr/local/ilogtail/ilogtail_config.json
  {
      "config_server_address" : "http://logtail.cn-hangzhou-intranet.log.aliyuncs.com",
      "data_server_list" :
      [
          {
              "cluster" : "cn-hangzhou",
              "endpoint" : "cn-hangzhou-intranet.log.aliyuncs.com"
          }
      ],
      "cpu_usage_limit" : 0.4,
      "mem_usage_limit" : 100,
      "max_bytes_per_sec" : 2097152,
      "process_thread_count" : 1,
      "send_request_concurrency" : 4,
      "streamlog_open" : false
  }

User identifier file

This file stores an Alibaba Cloud account ID that authorizes log collection from the server. Configure a user identifier.

• File path

  • Server environment

    • Linux: /etc/ilogtail/users/

    • Windows: C:\LogtailData\users\

  • Container environment

    The user identifier is stored in the ALIYUN_LOGTAIL_USER_ID environment variable of the Logtail container. Run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_ID command to view it.

• File example

  $ls /etc/ilogtail/users/

Custom identifier file (user_defined_id)

The user_defined_id file configures a custom identifier for machine groups. Create a custom identifier-based machine group.

• File path

  • Host environment

    • Linux: /etc/ilogtail/user_defined_id

    • Windows: C:\LogtailData\user_defined_id

  • Container environment

    The custom identifier is configured in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view this variable.

• File example

  $cat /etc/ilogtail/user_defined_id
  aliyun-ecs-rs1e16355

Logtail configuration file (user_log_config.json)

The user_log_config.json file stores collection configurations obtained from Simple Log Service. This JSON file updates whenever the collection configuration changes. To verify delivery, check whether the file exists and its content matches the configuration on Simple Log Service.

• File path

  • Host environment

    Operating system	Logtail	File path
    Linux	Logtail (64-bit)	/usr/local/ilogtail/user_log_config.json
    Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\user_log_config.json
    	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\user_log_config.json Note A 64-bit Windows operating system can run both 32-bit and 64-bit applications. For compatibility, the operating system stores 32-bit applications in a separate x86 directory.
    Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\user_log_config.json

  • Container environment

    The user_log_config.json file is stored in the Logtail container. The file path is /usr/local/ilogtail/user_log_config.json.

• File example

  $cat /usr/local/ilogtail/user_log_config.json
  {
     "metrics" : {
        "##1.0##k8s-log-c12ba2028*****939f0b$app-java" : {
           "aliuid" : "16542189*****50",
           "category" : "app-java",
           "create_time" : 1534739165,
           "defaultEndpoint" : "cn-hangzhou-intranet.log.aliyuncs.com",
           "delay_alarm_bytes" : 0,
           "enable" : true,
           "enable_tag" : true,
           "filter_keys" : [],
           "filter_regs" : [],
           "group_topic" : "",
           "local_storage" : true,
           "log_type" : "plugin",
           "log_tz" : "",
           "max_send_rate" : -1,
           "merge_type" : "topic",
           "plugin" : {
              "inputs" : [
                 {
                    "detail" : {
                       "IncludeEnv" : {
                          "aliyun_logs_app-java" : "stdout"
                       },
                       "IncludeLable" : {
                          "io.kubernetes.container.name" : "java-log-demo-2",
                          "io.kubernetes.pod.namespace" : "default"
                       },
                       "Stderr" : true,
                       "Stdout" : true
                    },
                    "type" : "service_docker_stdout"
                 }
              ]
           },
           "priority" : 0,
           "project_name" : "k8s-log-c12ba2028c*****ac1286939f0b",
           "raw_log" : false,
           "region" : "cn-hangzhou",
           "send_rate_expire" : 0,
           "sensitive_keys" : [],
           "tz_adjust" : false,
           "version" : 1
        }
     }
  }

AppInfo record file (app_info.json)

The app_info.json file records information about Logtail, such as its startup time, retrieved IP address, and hostname.

• File path

  • Host environment

    Operating system	Logtail	app_info.json file path
    Linux	Logtail (64-bit)	/usr/local/ilogtail/app_info.json
    Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\app_info.json
    	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json Note A 64-bit Windows operating system can run both 32-bit and 64-bit applications. For compatibility reasons, Windows stores 32-bit applications in a separate x86 directory.
    Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\app_info.json

  • Container environment

    The app_info.json file is stored in the Logtail container at /usr/local/ilogtail/app_info.json.

• File example

  $cat /usr/local/ilogtail/app_info.json
  {
     "UUID" : "",
     "hostname" : "logtail-ds-slpn8",
     "instance_id" : "E5F93BC6-B024-11E8-8831-0A58AC14039E_1**.***.***.***_1536053315",
     "ip" : "1**.***.***.***",
     "logtail_version" : "0.16.13",
     "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
     "update_time" : "2018-09-04 09:28:36"
  }

  Field	Description
  UUID	The server serial number.
  hostname	The server's hostname.
  instance_id	A randomly generated unique identifier for Logtail.
  ip	The IP address that Logtail retrieved. An empty field indicates that Logtail failed to retrieve an IP address and cannot run. In this case, you must assign an IP address to the server and restart Logtail. Note If you created an IP address-based machine group, ensure that the IP address configured in the machine group matches the one shown here. If they do not match, confirm the correct IP address of the server, and then either update the IP address in the machine group on the Simple Log Service console or modify the value of the working_ip parameter in Set Logtail startup parameters.
  logtail_version	The installed Logtail version.
  os	Operating system details, including the name, version, and architecture.
  update_time	The timestamp of the last Logtail startup.

Logtail operational log (ilogtail.LOG)

The ilogtail.LOG file records the operational logs of Logtail. The log levels, in ascending order, are INFO, WARN, and ERROR. You can ignore logs at the INFO level.

• File path

  • Server environment

    Operating system	Logtail	File path
    Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail.LOG
    Windows (64-bit operating system)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG
    	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail.LOG Note A 64-bit Windows operating system can run both 32-bit and 64-bit applications, storing 32-bit applications in a separate x86 directory for compatibility.
    Windows (32-bit operating system)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG

  • Container environment

    The Logtail container stores the ilogtail.LOG file at /usr/local/ilogtail/ilogtail.LOG.

• File Example

  $tail /usr/local/ilogtail/ilogtail.LOG
  [2018-09-13 01:13:59.024679]    [INFO]    [3155]    [build/release64/sls/ilogtail/elogtail.cpp:123]    change working dir:/usr/local/ilogtail/
  [2018-09-13 01:13:59.025443]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:175]    load logtail config file, path:/etc/ilogtail/conf/ap-southeast-1/ilogtail_config.json
  [2018-09-13 01:13:59.025460]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:176]    load logtail config file, detail:{
     "config_server_address" : "http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
     "data_server_list" : [
        {
           "cluster" : "ap-southeast-1",
           "endpoint" : "ap-southeast-1-intranet.log.aliyuncs.com"
        }
  ]

Logtail plugin log (logtail_plugin.LOG)

The logtail_plugin.LOG file records the operation logs for the Logtail plugin. The log levels, in ascending order, are INFO, WARN, and ERROR. You can ignore logs at the INFO level.

Use the logtail_plugin.LOG file to investigate collection errors, such as CANAL_RUNTIME_ALARM.

• File path

  • Host environment

    Operating system	Logtail	Path
    Linux	Logtail (64-bit)	/usr/local/ilogtail/logtail_plugin.LOG
    Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG
    	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\logtail_plugin.LOG Note A 64-bit Windows operating system can run both 32-bit and 64-bit applications. For compatibility, the operating system stores 32-bit applications in a separate x86 directory.
    Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG

  • Container environment

    The logtail_plugin.LOG file is stored in the Logtail container. The file path is /usr/local/ilogtail/logtail_plugin.LOG.

• Example

  $tail /usr/local/ilogtail/logtail_plugin.LOG
  2018-09-13 02:55:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 02:55:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
  2018-09-13 03:00:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 03:00:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
  2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
  2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
  2018-09-13 03:04:26 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
  2018-09-13 03:04:27 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
  2018-09-13 03:05:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
  2018-09-13 03:05:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop

Container path mapping (docker_path_config.json)

The docker_path_config.json file is created only when you collect container logs. This JSON file records the path mappings between container files and host files.

If you receive a DOCKER_FILE_MAPPING_ALARM error when diagnosing collection errors, this indicates the Docker file mapping could not be added. You can use the docker_path_config.json file to troubleshoot the issue.

• File path

  /usr/local/ilogtail/docker_path_config.json

• File example

  $cat /usr/local/ilogtail/docker_path_config.json
  {
     "detail" : [
        {
           "config_name" : "##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$nginx",
           "container_id" : "df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10",
           "params" : "{\n   \"ID\" : \"df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10\",\n   \"Path\" : \"/logtail_host/var/lib/docker/overlay2/947db346695a1f65e63e582ecfd10ae1f57019a1b99260b6c83d00fcd1892874/diff/var/log\",\n   \"Tags\" : [\n      \"nginx-type\",\n      \"access-log\",\n      \"_image_name_\",\n      \"registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest\",\n      \"_container_name_\",\n      \"nginx-log-demo\",\n      \"_pod_name_\",\n      \"nginx-log-demo-h2lzc\",\n      \"_namespace_\",\n      \"default\",\n      \"_pod_uid_\",\n      \"87e56ac3-b65b-11e8-b172-00163f008685\",\n      \"_container_ip_\",\n      \"172.20.4.224\",\n      \"purpose\",\n      \"test\"\n   ]\n}\n"
        } 
     ],
     "version" : "0.1.0"
  }