Simple Log Service:Logtail (legacy agent)

How it works

Monitor log files

After installing Logtail on a server and creating a Logtail configuration in the Simple Log Service console, Simple Log Service sends the configuration to Logtail in real time. Logtail then starts to monitor files based on this configuration. It scans log directories and files layer by layer according to the log path and maximum directory monitoring depth specified in the Logtail configuration.

After applying a Logtail configuration to a machine group, unmodified log files are considered historical files. Logtail does not automatically collect data from historical files. The collection process starts only when a file is modified. To collect data from historical log files, see Import historical logs.

To ensure timely and stable log collection, Logtail registers event listeners for the directories being monitored (using Inotify on Linux) and polls them periodically.

Read log files

After Logtail detects updated log files, it reads them.

• When Logtail reads a log file for the first time, it reads up to 1,024 KB of data by default.

  • If the file is smaller than 1,024 KB, Logtail reads data from the beginning of the file.

  • If the file is larger than 1,024 KB, Logtail reads the last 1,024 KB of data from the file.

  Note

  Simple Log Service allows you to customize this initial read size.

  • In the console, modify the Initial Collection Size parameter in the Logtail configuration. For more information, see Advanced Configurations.

  • Using the API: Modify the tail_size_kb parameter in the Logtail configuration. For more information, see Advanced parameters.

• If Logtail has previously read the log file, it resumes reading from the last saved checkpoint.

• Logtail reads a maximum of 512 KB at a time. Therefore, ensure that each log entry is smaller than 512 KB. Otherwise, the log entry cannot be read correctly.

Process logs

After reading logs, Logtail splits them into lines, parses the content, and sets the time field.

• Split logs into lines

  If a beginning-of-line regular expression is specified in the Logtail configuration, Logtail uses it to split the raw data into multiple log entries. If no regular expression is specified, Logtail treats each line as a single log entry.

• Parse logs

  Logtail parses each log entry based on the collection mode defined in the Logtail configuration.

  Note

  Complex regular expressions can cause high CPU usage. Use efficient regular expressions.

  If a log fails to parse, its handling depends on whether the drop failed to parse logs feature is enabled in the Logtail configuration.

  • If you enable Drop Failed to Parse Logs, the log is directly dropped and an error message for the parsing failure is reported.

  • If you disable Drop failed to parse logs, the original logs that failed to be parsed are uploaded, where the key is raw_log and the value is the log content.

• Set the log time field

  • If no time field is configured, Logtail sets the log time to the time when it parses the log.

  • If a time field is configured:

    • If the timestamp recorded in the log is within 12 hours of the current time, Logtail extracts the time from the parsed log fields.

    • If the timestamp differs from the current time by more than 12 hours, Logtail discards the log and reports an error.

Filter logs

After processing, Logtail filters logs based on the filter settings in the Logtail configuration.

• If a filter configuration is not set in the Logtail configuration, logs are not filtered and the process proceeds to the next step.

• If a Logtail configuration has a filter configuration, Logtail traverses and validates all fields in each log.

  Logtail collects only logs that match the filter conditions.

Aggregate logs

To reduce network requests, Logtail caches logs for a short period after processing and filtering. It then aggregates them into batches before sending them to Simple Log Service. Logtail sends a batch of logs when one of the following conditions is met:

• The aggregation time exceeds 3 seconds.

• The number of aggregated logs exceeds 4,000.

• The total size of aggregated logs exceeds 512 KB.

Send logs

Logtail sends the aggregated logs to Simple Log Service. If sending fails, Logtail automatically retries or discards the data based on the error received.

Benefits

• Collects logs non-intrusively from log files. This process requires no modification to your application code and does not affect your application's runtime.

• Supports collecting various data types, including text logs, binlogs, HTTP data, and container logs.

• Collects data from container environments such as Docker and Kubernetes clusters.

  • Alibaba Cloud Container Service for Kubernetes (ACK): See Collect logs from Kubernetes.

  • Self-managed Kubernetes: See Collect logs from Kubernetes.

  • Self-managed Docker clusters: See Collect logs from Docker containers (stdout and files).

• Manages collection exceptions gracefully. In case of network issues or server-side errors, Logtail uses automatic retries and a local cache to ensure data security.

• After you install Logtail, you can manage all settings, such as machine groups and Logtail configurations, centrally from the Simple Log Service console.

• Logtail operates under strict CPU, memory, and network usage limits to prevent it from impacting the performance of other services on your server.

Configuration process

1. Install Logtail.

2. Create a machine group.

   A Simple Log Service project supports using IP addresses or custom identifiers to create machine groups.

3. Create a Logtail configuration.

   You can use the configuration wizard in the Simple Log Service console to complete the setup. For more information, see Collect text logs and Collect container logs.

After completing these steps, Logtail starts to collect logs from your servers and send them to the specified logstore. You can then query the logs by using the Simple Log Service console, API, SDK, or CLI.

Key concepts

• Machine group: A machine group contains one or more servers from which you want to collect the same type of logs. When you apply a Logtail configuration to a machine group, Simple Log Service uses the configuration to collect logs from all servers in the group.

  Simple Log Service uses machine groups to manage all servers that use Logtail for log collection. You can define a machine group by using IP addresses or a custom identifier. You can manage machine groups in the Simple Log Service console, which includes operations such as creating and deleting groups, or adding and removing servers. For more information, see Machine groups.

• Logtail: A log collection agent provided by Simple Log Service that runs on the server from which logs are to be collected.

  • On Linux operating systems, Logtail is installed in the /usr/local/ilogtail directory and starts two independent processes whose names begin with ilogtail: a collection process and a daemon. The runtime logs are saved in the /usr/local/ilogtail/ilogtail.LOG file. For more information, see Install Logtail (Linux).

  • Windows

    • Logtail (32-bit)

      • When installed on a 32-bit Windows operating system, the installation directory is C:\Program Files\Alibaba\Logtail.

      • When installed on a 64-bit Windows operating system, the installation directory is C:\Program Files (x86)\Alibaba\Logtail.

        Note

        A 64-bit Windows OS can run both 32-bit and 64-bit applications. For compatibility, Windows uses a separate x86 directory to store 32-bit applications on a 64-bit OS.

    • Logtail (64-bit)

      Logtail is supported only on 64-bit Windows operating systems. The installation directory is C:\Program Files\Alibaba\Logtail.

    You can view the LogtailDaemon service (for Logtail 1.0.0.0 and later) or the LogtailWorker service (for Logtail 0.x.x.x versions) in Control Panel > Administrative Tools > Services to check the running status of Logtail. The program running logs are saved in the ilogtail.LOG file in the installation directory. For more information, see Install Logtail on Windows.

• Logtail configuration: A set of policies that Logtail uses to collect logs. You can create a custom collection policy by setting parameters such as the data source and collection mode. A Logtail configuration defines how to collect, parse, and send logs of the same type from a server to a specified logstore in Simple Log Service.

Features

Data collection reliability

During log collection, Logtail periodically saves its collection progress, known as a checkpoint, to a local file. If an unexpected shutdown or process crash occurs, Logtail restarts and resumes collection from the last saved checkpoint to minimize data loss. Logtail operates based on its startup configuration file. If its resource usage exceeds the specified limits for more than 5 minutes, Logtail forcefully restarts. This may result in some duplicate data being collected.

References

• Logtail diagnostics

• Logtail FAQ

• Logtail limits