Simple Log Service:Logtail (legacy)

Collection process

Monitor logs

After you install Logtail on servers and create a Logtail configuration in the Simple Log Service console, the configuration is delivered to servers in real time. Logtail then monitors log files by scanning directories based on the configured log file path and maximum directory monitoring depth.

Log files not updated after a Logtail configuration is applied are considered historical and are not collected. Updated files are read and sent to Simple Log Service. For more information about collecting historical log files, see Import historical logs from log files.

Logtail registers event listeners that poll log directories at regular intervals for timely collection. On Linux, Inotify monitors directories and polls log files.

Read logs

After detecting updated log files, Logtail reads data in the log files.

• On first read, Logtail reads up to 1,024 KB by default.

  • If the file is smaller than 1,024 KB, Logtail reads from the beginning.

  • If the file is larger than 1,024 KB, Logtail reads the last 1,024 KB.

  Note

  The initial read size is configurable.

  • Console: Modify First Collection Size in the Logtail Config page. Advanced settings.

  • API: Modify tail_size_kb in the Logtail configuration. advanced.

• If a log file was previously read, Logtail resumes from the last checkpoint.

• Logtail reads up to 512 KB per read. Each log entry must not exceed 512 KB.

Process logs

Logtail splits each log into lines, parses the content, and sets the time field.

• Split a log into multiple lines

  Logtail splits logs into multiple lines based on a first-line regex you specify. Without a regex, each line is treated as a separate log.

• Parse logs

  Logtail parses each log based on the collection mode specified in the Logtail configuration.

  Note

  Complex regular expressions can cause high CPU usage. Use efficient expressions to minimize resource consumption.

  If Logtail fails to parse a log, it handles the failure based on the Drop Failed to Parse Logs setting.

  • If Drop Failed to Parse Logs is enabled, Logtail drops the log and reports an error.

  • If Drop Failed to Parse Logs is disabled, Logtail uploads the log with the key raw_log and the value set to the log content.

• Configure the time field for a log

  • If you do not configure the time field, the log time defaults to the parse time.

  • If you configure the time field, processing depends on the time difference:

    • Within 12 hours: the log time is extracted from the parsed log fields.

    • Beyond 12 hours: the log is dropped and an error is reported.

Filter logs

After processing, Logtail filters logs based on the specified conditions.

• If no conditions are set in Filter Configuration, logs are not filtered.

• If conditions are set in Filter Configuration, each log's fields are evaluated.

  Only logs that match the filter conditions are collected.

Aggregate logs

To reduce network requests, Logtail caches processed and filtered logs and sends them in batches when any of the following thresholds is reached:

• The aggregation duration exceeds 3 seconds.

• The number of aggregated logs exceeds 4,000.

• The total size of aggregated logs exceeds 512 KB.

Send logs

Logtail sends aggregated logs to Simple Log Service. On failure, Logtail retries or drops the packet based on the HTTP status code.

Benefits

• Non-intrusive file-based collection. No code changes required, and collection does not affect application performance.

• Collects text logs, binary logs, HTTP logs, and container logs.

• Collects logs from container clusters such as Docker and Kubernetes.

• Handles collection exceptions automatically. On network or server failures, Logtail retries and caches data locally.

• Centralized management through Simple Log Service. Install Logtail, create a machine group and configuration, and collection starts automatically.

• Built-in resource protection. Simple Log Service enforces strict CPU, memory, and network limits to prevent Logtail from affecting other services on the server.

Configuration process

1. Install Logtail.

2. Create a machine group.

   Create an IP address-based machine group or a custom identifier-based machine group for a Simple Log Service project.

3. Create a Logtail configuration.

   Use the Simple Log Service console to collect text logs or collect container logs.

After setup, Logtail collects logs from your servers and sends them to the specified Logstore. Query logs through the Simple Log Service console, API, SDK, or CLI.

Terms

• Machine group: A group of one or more servers that collect the same type of logs. When a Logtail configuration is applied, Simple Log Service collects logs from all servers in the group.

  Define machine groups by IP address or custom identifier. Manage them in the Simple Log Service console. Overview.

• Logtail: The log collection agent provided by Simple Log Service, running on each server from which logs are collected.

  • Linux: Logtail is installed in /usr/local/ilogtail and runs two processes (collector and daemon) starting with ilogtail. Operational logs are stored in /usr/local/ilogtail/ilogtail.LOG. For more information, see Install Logtail on a Linux server.

  • Windows:

    • Logtail (32-bit)

      • 32-bit Windows: C:\Program Files\Alibaba\Logtail.

      • 64-bit Windows: C:\Program Files (x86)\Alibaba\Logtail.

        Note

        64-bit Windows stores 32-bit applications in a separate x86 directory for compatibility.

    • Logtail (64-bit)

      64-bit Logtail is supported only on 64-bit Windows. Installation directory: C:\Program Files\Alibaba\Logtail.

    To check Logtail status, go to Control Panel > Administrative Tools > Services. For V1.0.0.0 or later, check the LogtailDaemon service. For V0.x.x.x, check the LogtailWorker service. Operational logs are in ilogtail.LOG in the installation directory. For more information, see Install Logtail on a Windows server.

• Logtail configurations: Policies that define how Logtail collects and parses logs. Each configuration specifies the data source, collection mode, and target Logstore.

Basic features

Data collection reliability

Logtail periodically saves checkpoints locally. After an unexpected shutdown or crash, it resumes from the last checkpoint to minimize data loss. If resource usage exceeds configured limits for more than 5 minutes, Logtail is forcefully restarted, which may cause duplicate collection.

References

• Logtail diagnostics

• Logtail FAQ

• Logtail limits