Use Logtail to collect data - Log Service - Alibaba Cloud Documentation Center

Logtail is a log collection agent that is provided by Log Service. You can use Logtail to collect logs from multiple data sources in real time. These sources include Alibaba Cloud Elastic Compute Service (ECS) instances, data centers, and servers that belong to third-party cloud service providers. This topic describes the features, benefits, limits, and configuration process of Logtail.

Configuration process

Install Logtail on a server.
- For more information about how to install Logtail on Linux, see Install Logtail on a Linux server.
- For more information about how to install Logtail on Windows, see Install Logtail on a Windows server.
The server on which you want to install Logtail can be an ECS instance that does not belong to the current account, or belongs to a local data center or third-party cloud service provider. In this case, you must specify a user identity for the server.
For more information, see Configure a user identifier.
Create a machine group.
- For more information about how create an IP address-based machine group, see Create an IP address-based machine group.
- For more information about how to create a custom identifier-based machine group, see Create a custom identifier-based machine group.
Create Logtail configurations and apply the configurations to the machine group.
You can perform the preceding operations in the Log Service console. For more information, see Overview.

After you perform the preceding operations, Logtail collect logs from you server and send the logs to the specified Logstore. You can query logs by using the Log Service console, API operations, SDKs, or CLI.

Benefits

Supports non-intrusive log collection based on log files. You do not need to modify your application code. Your applications are not affected when Logtail collects logs.
Allows you to collect text logs, binary logs, HTTP data, and container logs.
Allows you to collect logs from standard containers and Kubernetes clusters.
- For more information about how to collect logs from Container Service for Kubernetes, see Overview.
- For more information about how to collect logs from self-managed Kubernetes clusters, see Overview.
- For more information about how to collect logs from self-managed Docker clusters, see Collect logs from standard Docker containers.
Handles exceptions during log collection. If a network or server exception occurs, Logtail retries log collection and caches logs on local servers to ensure data security.
Provides centralized management based on Log Service. After you install Logtail on servers and create a machine group and Logtail configurations, Logtail collects logs from the servers and sends the logs to Log Service.
Provides a comprehensive self-protection mechanism. The CPU, memory, and network resources that Logtail can use are limited. This ensures that Logtail does not affect the performance of other services on the server.

Limits

For more information about the limits of Logtail, see Logtail limits.

Terms

Machine group: A machine group contains one or more servers from which logs of a specific type are collected. After you apply Logtail configurations to a machine group, Log Service collects logs from the servers in the machine group based on the configurations.
You can set an IP address-based identifier or a custom identifier for a machine group. Then, you can manage the servers in the machine group based on the identifier. You can create and delete a machine group, add servers to a machine group, and remove servers from a machine group in the Log Service console.
Logtail is a log collection agent that is provided by Log Service. Logtail runs on servers from which you want to collect logs.
- For a Linux-based server, Logtail is installed in the /usr/local/ilogtail directory. Logtail initiates the following separate processes whose names start with ilogtail: a log collection process and a daemon process. The logs of Logtail are stored in the /usr/local/ilogtail/ilogtail.LOG directory.
- For a Windows-based server, Logtail is installed in the C:\Program Files\Alibaba\Logtail directory (32-bit system) or C:\Program Files (x86)\Alibaba\Logtail directory (64-bit system). Choose Control Panel > Administrative Tools > Services. On the Services window, you can view the LogtailDaemon service. The logs of Logtail are stored in the ilogtail.LOG file.
Logtail configurations for log collection: Logtail configurations for log collection are a set of policies that Logtail uses to collect logs. You can specify the data source and collection mode to create custom Logtail configurations for log collection. The configurations specify how to collect logs from servers, parse the logs, and send the logs to a specified Logstore.

Features


Feature	Description
Log collection in real time	Logtail monitors log files, and reads and parses incremental logs in real time. In most cases, logs are sent to Log Service within 3 seconds after the logs are generated. For more information, see Log collection process of Logtail. Note Logtail does not collect historical data. If a log entry is generated for more than 12 hours before the log is read, Logtail does not collect the log entry.
Automatic log rotation	Multiple applications rotate log files based on the file size or date. The original log file is renamed and an empty log file is created during the rotation process. For example, the app.LOG file is renamed app.LOG.1 and app.LOG.2 during log rotation. You can specify the file to which collected logs are written, for example, app.LOG. Logtail monitors the log rotation process to ensure that no logs are lost.
Multiple data sources	Logtail can collect text logs, syslogs, HTTP logs, and MySQL binlogs. For more information, see Data collection overview.
Compatibility with open source collection agents	You can use open source agents such as Logstash and Beats to collect data. Then, you can use Logtail to collect data from the agents and send the data to Log Service. For more information, see Data collection overview.
Automatically handle collection exceptions	If data fails to be sent to Log Service due to exceptions, Logtail retries to collect logs based on the scenario. The exceptions include server errors, network errors, and quota exhaustion. If the retry fails, Logtail writes the data to the local cache and resends the data after 3 seconds. For more information, see How do I use the automatic diagnostic tool of Logtail?.
Flexible configurations	Logtail allows you to create configurations for log collection in a flexible manner. You can specify the directories and files from which logs are collected. You can also specify an exact match or a wildcard match based on your business requirements. You can also specify the log collection mode and customize the fields that you want to extract. You can use a regular expression to extract fields from logs. Log data in Log Service must have the timestamp information. Logtail allows you to customize log time formats and then extract the required timestamps from the time information based on different formats.
Automatically synchronize Logtail configurations	After you create or update Logtail configurations for log collection in the Log Service console, the configurations are synchronized to the servers in which Logtail is installed and take effect within 3 minutes. Logs are collected based on the original configurations during the synchronization.
Status monitoring	Logtail monitors the CPU and memory resources that are consumed in real time. This ensures that Logtail does not consume an excessive number of resources or affect other services. If the resource consumption exceeds the limit, Logtail is automatically restarted. Logtail also monitors the network bandwidth resources that are consumed. This ensures that Logtail does not consume an excessive amount of bandwidth. For more information, see Startup configuration file (ilogtail_config.json).
Signature and encryption	Logtail retrieves the AccessKey pair of your Alibaba Cloud account and uses the pair to sign all log data that is sent to Log Service. This way, data tampering is prevented during data transmission. Note Logtail retrieves the AccessKey pair of your Alibaba Cloud account by using the HTTPS protocol to ensure the security of your AccessKey pair.

Data collection reliability

Logtail stores checkpoints that are periodically collected to the local server during log collection. If an exception such as an unexpected server shutdown or a process failure occurs, Logtail restarts and then collects data from the last checkpoint. This process avoids incomplete data collection. Logtail runs based on the startup parameters that are specified in the startup configuration file. If the usage of a resource exceeds the limit for more than 5 minutes, Logtail is forcibly restarted. After the restart, a small amount of duplicate data may be collected to the specified Logstore.

To improve log collection reliability, Logtail uses multiple internal mechanisms. However, logs may fail to be collected in the following scenarios:

Logtail is not running, but logs are rotated multiple times.
The log rotation rate is high, for example, one rotation per second.
The log collection rate is lower than the log generation rate for a long period of time.