Logtail is a log collection agent that is provided by Simple Log Service. You can use Logtail to collect logs from multiple data sources, including Alibaba Cloud Elastic Compute Service (ECS) instances, servers in data centers, and servers from third-party cloud service providers. This topic describes the features, benefits, limits, and configuration process of Logtail.
Configuration process
Install Logtail on a server.
For more information about how to install Logtail on an ECS instance, see Install Logtail on ECS instances.
For more information about how to install Logtail on a Linux server, see Install Logtail on a Linux server.
For more information about how to install Logtail on a Windows server, see Install Logtail on a Windows server.
Configure a user identifier for the server. This operation is required if your server is an ECS instance that belongs to a different Alibaba Cloud account, a server in a data center, or a server from a third-party cloud service provider.
For more information, see Configure a user identifier.
Create a machine group.
For more information about how to create an IP address-based machine group, see Create an IP address-based machine group.
For more information about how to create a custom identifier-based machine group, see Create a custom identifier-based machine group.
Create a Logtail configuration and apply the Logtail configuration to the machine group.
You can perform the preceding operations in the Simple Log Service console. For more information, see collect text logs and collect container logs.
After you perform the preceding operations, Logtail collects logs from your server and sends the logs to a specified Logstore. You can query the logs by using the console, API, SDK, or CLI of Simple Log Service.
Benefits
Supports non-intrusive log collection based on log files. You do not need to modify your application code, and log collection does not affect the operation of your applications.
Collects text logs, binary logs, HTTP logs, and container logs.
Collects logs from standard containers and various container clusters, such as Kubernetes clusters.
For more information about how to collect logs from Container Service for Kubernetes (ACK) clusters, see Overview.
For more information about how to collect logs from user-created Kubernetes clusters, see Overview.
For more information about how to collect logs from user-created Docker clusters, see Collect logs from standard Docker containers.
Handles exceptions that occur in the log collection process. If issues such as network or server exceptions occur, Logtail retries log collection and caches data locally to ensure data security.
Provides centralized management based on Simple Log Service. After you install Logtail on a server from which you want to collect logs and create a machine group and Logtail configuration, Logtail collects logs from the server.
Provides a comprehensive self-protection mechanism. To ensure that Logtail does not significantly affect the performance of other services that run on the same server as Logtail, Simple Log Service limits the CPU, memory, and network resources that can be used by Logtail and provides a self-protection mechanism.
Processing capabilities and limits
For more information, see Logtail limits.
Terms
Machine group: A machine group contains one or more servers from which a specific type of logs are collected. After you apply a Logtail configuration to a machine group, Simple Log Service collects logs from all servers in the machine group based on the Logtail configuration.
Simple Log Service uses machine groups to manage all servers from which you want to collect logs by using Logtail. You can define a machine group based on an IP address or a custom identifier. You can manage machine groups in the Simple Log Service console. For example, you can create or delete a machine group and add a server to or remove a server from a machine group. For more information, see Introduction.
Logtail: Logtail is a log collection agent that is provided by Simple Log Service. Logtail runs on servers from which you want to collect logs.
Linux: In Linux, Logtail is installed in the /usr/local/ilogtail directory and initiates two independent processes whose names start with ilogtail. One is a collection process and the other is a daemon. The program operational logs are stored in the /usr/local/ilogtail/ilogtail.LOG file. For more information, see Install Logtail on a Linux server.
Windows:
Logtail (32-bit)
In 32-bit Windows, Logtail is installed in the C:\Program Files\Alibaba\Logtail directory.
In 64-bit Windows, Logtail is installed in the C:\Program Files (x86)\Alibaba\Logtail directory.
NoteYou can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
Logtail (64-bit)
You can install Logtail (64-bit) only in 64-bit Windows. The installation directory is C:\Program Files\Alibaba\Logtail.
To check the status of Logtail, you can perform the following operations: Choose . If you install Logtail V1.0.0.0 or later, view the LogtailDaemon service. If you install Logtail V0.x.x.x, view the LogtailWorker service. The program operational logs are stored in the
ilogtail.LOGfile of the installation directory. For more information, see Install Logtail on a Windows server.
Logtail configuration: A Logtail configuration is a set of policies used by Logtail to collect logs. You can specify the data source and collection mode to create custom Logtail configurations for log collection. A Logtail configuration is used to collect a specific type of logs from servers, parse the collected logs, and send the logs to a specified Logstore of Simple Log Service.
Features
Feature | Description |
Real-time log collection | Logtail dynamically monitors log files and reads and parses incremental logs in real time. In most cases, logs are sent to Simple Log Service within 3 seconds after the logs are generated. For more information, see Log collection process of Logtail. Note Logtail does not collect historical logs. Logs that are read 12 hours or later after the logs are generated are discarded. For more information about how to collect logs from historical log files, see Import historical logs from log files. |
Automatic log rotation | Multiple applications rotate log files based on the file size or date. In the rotation process, original log files are renamed and new empty log files are created. For example, files such as app.LOG.1 and app.LOG.2 are generated for the app.LOG file after log rotation. You can specify the file to which collected logs are written. Example: app.LOG. Logtail automatically monitors the log rotation process and ensures that no logs are lost during this process. |
Support for multiple data sources | Logtail can collect text logs, syslogs, HTTP logs, and MySQL binary logs. For more information, see Data collection overview. |
Compatibility with an open-source collection agent | Logtail can collect data that is collected by using open-source software such as Logstash and Beats to Simple Log Service. For more information, see Data collection overview. |
Automatic handling of collection exceptions | If data transmission fails due to an exception such as Simple Log Service errors, network errors, or quota exhaustion, Logtail actively retries log collection based on the specific scenario. If the retry fails, Logtail writes the data to its local cache and sends the data again after 3 seconds. For more information, see How do I use the automatic diagnostic tool of Logtail? |
Flexible collection configuration | You can collect logs in a flexible manner based on Logtail configurations. You can specify the directories and files from which logs are collected. Exact match and wildcard match are supported. You can specify the log collection mode and the fields that you want to extract. You can use a regular expression to extract logs. The log data models of Simple Log Service require that each log have a precise timestamp. Logtail supports custom log time formats, which allows you to extract the required timestamp information from log data of different formats. |
Automatic synchronization of Logtail configurations | After you create or update a Logtail configuration in the Simple Log Service console, Logtail automatically receives and applies the configuration within 3 minutes in most cases. No logs are lost during the Logtail update process. |
Status monitoring | Logtail monitors its CPU and memory consumption in real time. This helps prevent Logtail from consuming excessive resources. The overconsumption of resources may affect other services that run on the same server as Logtail. If the resource usage of Logtail exceeds the limit, Logtail automatically restarts. If the network bandwidth usage exceeds the limit, Logtail triggers throttling. For more information, see Startup configuration file (ilogtail_config.json). |
Transmission of signed data | To prevent data from being tampered with during data transmission, Logtail obtains a private token from Simple Log Service over a trusted channel and signs all log data packets that are sent. Note Logtail obtains a private token over HTTPS to ensure the security of your token. |
Data collection reliability
During data collection, Logtail stores the collected checkpoint information to a local server on a regular basis. If an exception such as unexpected shutdown of a server occurs or a process unexpectedly exits, Logtail collects data from the last recorded checkpoint after it is restarted. This prevents data loss. Logtail runs based on the startup parameters that are specified in the startup configuration file. If the resource usage of Logtail exceeds a limit for more than 5 minutes, Logtail is forcefully restarted. Duplicate data may be collected after the restart.
Logtail uses internal mechanisms to improve log collection reliability. However, logs may be lost in the following situations:
Logtail is not running, but log files are rotated multiple times.
The rotation rate of log files is extremely high, such as one rotation per second.
The log collection rate is lower than the log generation rate for a long period of time.