How to use the Logtail diagnostics feature - Simple Log Service

When you use Logtail to collect logs, you may encounter issues such as regular expression parsing failures, incorrect file paths, or traffic exceeding the Shard service capacity. Simple Log Service provides a diagnostics feature to help you quickly locate Logtail collection errors.

Prerequisites

Logs are collected using Logtail. For more information, see Data collection.
Important log collection is enabled for the project that the Logtail configuration belongs to. For more information, see Enable service logs.

Background information

There are two ways to diagnose Logtail issues: console diagnostics and the Logtail automatic diagnostic tool.

Console diagnostics includes advanced and basic diagnostics:
- Advanced diagnostics (recommended): provides a dashboard to view and query Logtail exception details, including those from the past.
  Important
  The dashboard of the advanced diagnostics depends on the important logs in the project. For more information, see Service log types.
- Basic diagnostics: only provides information about the exceptions of the previous hour.
The Logtail automatic diagnostic tool is only supported on Linux servers and requires the curl tool for network connectivity checks. The process is as follows:

Console diagnostics

Log on to the Simple Log Service console.
In the Projects section, click the one you want to manage.
In the Log Storage > Logstores tab, hover your cursor over the target logstore and click the icon.
View diagnostic information.
Important
To use the advanced diagnostics, make sure that important log collection is enabled. For more information, see Important logs.
Basic diagnostics
The Log Collection Error panel displays all Logtail collection errors for the logstore. Click an error code to view the details of the error. For more information, see Common error types of data collection in Simple Log Service.
Advanced diagnostics
On the LoongCollector/Logtail Exception Monitoring page, view information such as active Logtail count and all error messages. For more information, see View data reports and Common error types of data collection in Simple Log Service.
After resolving issues, check for remaining errors. Historical errors are shown until expired. You can ignore them and only confirm new errors after fixing previous ones. Logtail reports errors every 10 minutes.
Note
To see full logs discarded due to parsing failures, access your Logtail server or container and check the operational logs. These logs are stored in the following paths:
- Server scenario: in the /usr/local/ilogtail/ilogtail.LOG file on the server where Logtail is installed.
- Container scenario: in the /usr/local/ilogtail/ilogtail.LOG file in the container where Logtail is installed.

Logtail automatic diagnostic tool

Log on to the Linux server where Logtail is installed and run the following command to download the tool.
```
wget http://logtail-release.oss-cn-hangzhou.aliyuncs.com/linux64/checkingtool.sh -O checkingtool.sh
```

Run the following command to execute the tool.

chmod 744 ./checkingtool.sh
./checkingtool.sh
sh checkingtool.sh

Select a language.

[Info]:     Logtail checking tool version : 0.3.1
[Input]:  Please choose language Please select language : 
                1. Chinese. 
                2. English. 
        Language :

Select a check process.

[Input]:  please choose which type you want to check : 
                1. MachineGroup heartbeat fail. 
                2. MachineGroup heartbeat is ok, but log files have not been collected. 
        Type :

1: If the heartbeat status of the machine group is abnormal, select this option.
2: If the heartbeat status of the machine group is normal but log files are not collected, select this option.

Check the heartbeat status of the machine group

If you select option 1, the tool performs the following checks:

Check basic information.

Logtail is installed.
Logtail is running.
The status of Secure Sockets Layer (SSL) encryption is normal.
Network connectivity exists between the server and Simple Log Service.

[Info]:     Check Logtail install files
[Info]:     Install file: ilogtail_config.json exists.                          [ OK ]
[Info]:     Install file: /etc/init.d/ilogtaild exists.                         [ OK ]
[Info]:     Install file: ilogtail exists.                                      [ OK ]
[Info]:     Bin file: /usr/local/ilogtail/ilogtail_2.0.8 exists.                [ OK ]
[Info]:     Logtail version : 2.0.8                                             [ OK ]

[Info]:     Check Logtail running status
[Info]:     Logtail is runnings.                                                [ OK ]
[Info]:     Check network status
[Info]:     Logtail is using ip: 172.31.0.246
[Info]:     Logtail is using UUID: D1BA8F5A-FA52-11EF-8935-00163E4C****
[Info]:     Check SSL status
Redirecting to /bin/systemctl status iptables.service
Unit iptables.service could not be found.
[Info]:     SSL status OK.                                                      [ OK ]
[Info]:     Check logtail config server
[Info]:     config server address: http://logtail.cn-hangzhou-intranet.log.aliyuncs.com
[Info]:     Logtail config server HTTP OK                                       [ OK ]
[Info]:     Logtail config server HTTPS OK                                      [ OK ]

Check whether your server is an Alibaba Cloud ECS instance and belongs to the same account as the current project.

If yes, enter y to continue to the next step.

Otherwise, enter N. The tool outputs the locally configured user identifier information. If your Alibaba Cloud account ID is not included, reconfigure the user identifier.

[Input]:  Is your server Alibaba Cloud ECS and belong to the same account of the current SLS Project ? (y/N)n
[Info]:     Check aliyun user id(s)
[Info]:     aliyun user id : 198********815 .                                 [ OK ]

[Input]:  Is your project owner account ID among the above IDs ? (y/N)n
[Error]:    please configure user IDs (account IDs), this doc will help
            [ https://www.alibabacloud.com/help/doc-detail/49007.htm ]          [ Error ]

Check whether the region of your project is the same as the region you selected when you install Logtail.
- If yes, enter y to continue to the next step.
- Otherwise, enter N. The tool prompts you to reinstall Logtail or recreate the project to ensure that the project region and Logtail installation region are the same. For more information, see Install Logtail on a Linux server.
```
[Input]:  please make sure your project is in this region : { cn-hangzhou } (y/N) :n
[Error]:    please reinstall your Logtail or recreate your project to
            make sure your project is located in the same region with your Logt [ Error ]
```
Check whether the IP address or custom ID in your machine group matches the one in the returned message.
- If yes, enter y. The tool generates a report. If the issue persists, copy the output report content and submit a ticket.
- Otherwise, enter N. The tool prompts you to correct the IP address or custom ID in the machine group. For more information, see Modify the machine group information.
```
[Input]:  please make sure your machine group's ip is same with : { 172.31.0.246 } or your machine group's userdefined-id is in : { user-defined-linux-1 } (y/N) :n
[Error]:    please correct your machine group's ip or userdefined-id,
            save the configuration, wait 1 minute and check again               [ Error ]
```

Check whether errors occur when Logtail collects logs

If you select option 2, the tool performs the following checks:

Ensure the machine group's heartbeat status is normal and its IP address matches the one in the returned message.
- If yes, enter y to continue to the next step.
- Otherwise, enter N. The tool prompts you to return to check the heartbeat status of the machine group.
```
[Input]:  please make sure your machine's heartbeat is ok and machine group's ip is same with : { 172.31.0.246 } (y/N) :n
[Error]:    please rerun the script and choose type 1 to check machine group's  [ Error ]
```
Check whether your Logtail configuration is applied to the machine group.
- If yes, enter y to continue to the next step. For more information, see View machine group configurations.
- Otherwise, enter N. The tool prompts you to apply the Logtail configuration to the machine group.
```
[Input]:  please make sure you have applied collection config to the machine group (y/N) :n
[Error]:    please apply collection config to the machine group                 [ Error ]
```

Check whether the log file path in the Logtail configuration is valid.

You must enter the full path of the log file. If no file is matched, check whether the specified path can match the file.

If the configuration is incorrect, modify it, wait for about 1 minute, and then run the script again to check.

[Input]:  please input your log file's full path (eg. /var/log/nginx/access.log) : /var/log/nginx/access.log
[Info]:     Check specific log file
[Info]:     Check if specific log file [ /var/log/nginx/access.log ] is included by user log config.
[Info]:     Specific log file exists.                                           [ OK ]
[Error]:    No match config for your log file.                                  [ Error ]
[Suggestion]:  Please check collection config is created in SLS Project/Logstore and make sure you have applied config to your machine group

If the issue persists, copy the diagnostic report and submit a ticket.

[Input]:  please make sure all the check items above have passed. If the problem persists, please copy all the outputs and submit a ticket in the ticket system. : (y/N)y
[Info]:     Check system support
[Info]:     Check system support OK.                                            [ OK ]

[Info]:     Check Logtail install files
[Info]:     Install file: ilogtail_config.json exists.                          [ OK ]
[Info]:     Install file: /etc/init.d/ilogtaild exists.                         [ OK ]
[Info]:     Install file: ilogtail exists.                                      [ OK ]
[Info]:     Bin file: /usr/local/ilogtail/ilogtail_2.0.8 exists.                [ OK ]
[Info]:     Logtail version : 2.0.8                                             [ OK ]
......
......

Quick check

Collection failures often stem from incorrect user identifiers. Quickly check and output the configured Alibaba Cloud account ID and custom user identifier on the Logtail client. No alert occurs if these are missing. If configured, ensure they match your settings. Otherwise, reconfigure by referring to Machine groups.

Use the following command to perform a quick check:

./checkingtool.sh --logFile [LogFileFullPath]

If the specified log file passes the check and the Logtail runtime environment is normal, view the error logs of the related parameters in the Simple Log Service console. For more information, see Console diagnostics.

Common parameters of the diagnostic tool

Common parameter	Description
`--help`	View the help documentation.
`--logFile [LogFileFullPath]`	Check whether Logtail collects logs from the `LogFileFullPath` path. Then check the basic Logtail runtime environment, such as installation file integrity, running status, Alibaba Cloud account ID, and network connectivity.
`--logFileOnly [LogFileFullPath]`	Only check whether Logtail collects logs from the `LogFileFullPath` path.
`--envOnly`	Only check the runtime environment of Logtail.

Common issues with Logtail collection

The following table describes the causes of common Logtail collection errors and their solutions.

Note

Logtail only collects incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

Issue	Solution
Installation files are missing.	Reinstall Logtail.
Logtail is not running.	Run the `/etc/init.d/ilogtaild start` command to start Logtail.
Multiple Logtail processes exist.	Run the `/etc/init.d/ilogtaild stop` command to stop Logtail. Run the `/etc/init.d/ilogtaild start` command to start Logtail.
Port 443 is disabled.	Configure your firewall to enable port 443.
The configuration server cannot be found.	Check whether Logtail is correctly installed. If Logtail is not installed on the Linux server, rerun the installation command. For more information, see Install Logtail on a Linux server.
The user configuration does not exist.	Check whether the following operations are performed: A Logtail configuration is created. The server is included in a machine group. The Logtail configuration is applied to the machine group.
The specified log file cannot be found.	Check whether the Logtail configuration is valid.
The specified log file is matched by two or more Logtail configurations.	If the log file is matched by two or more Logtail configurations, Logtail randomly selects one. Use only one Logtail configuration to match the log file.