Troubleshoot issues related to Logtail machine groups - Simple Log Service

If no heartbeats are detected in a Logtail machine group when you use Logtail to collect logs, you can troubleshoot the error manually or by using the Logtail automatic diagnostic tool. A Logtail machine group contains servers on which Logtail is installed. This topic describes how to troubleshoot the error if a Logtail machine group has no heartbeats.

Troubleshooting process

If you use Logtail to collect logs, Logtail sends heartbeat packets to Log Service at a scheduled time after Logtail is installed on a server. If no heartbeats are detected in the machine group to which the server belongs, the connection between Logtail and Log Service fails. Log Service provides automatic and manual diagnostic methods. You can select a method based on your business requirements.

Automatic diagnostics: Log Service provides the Logtail automatic diagnostic tool. You can select this method only if Logtail is installed on Linux servers. For more information, see How do I use the Logtail automatic diagnostic tool?
Manual diagnostics: If the Logtail automatic diagnostic tool fails to identify the root cause of the error or Logtail is installed on a Windows server, perform the following steps.

Troubleshooting method

Step 1: Check whether Logtail is installed

You can view the status of Logtail to check whether Logtail is installed on a server.

View the status of Logtail.
- Linux:
  Run the following command to view the status of Logtail:
```
sudo /etc/init.d/ilogtaild status 
```
  If the following output is returned, Logtail is installed:
```
ilogtail is running
```
- Windows:
  1. Open the Run window and enter services.msc to open the Services window.
  2. View the running status of the LogtailDaemon service (Logtail V1.0.0.0 or later) or LogtailWorker service (Logtail V0.x.x.x).
    If the service is in the Running state, Logtail is installed.
Perform operations based on the status of Logtail.
- If Logtail is installed, perform the steps that are described in Step 2: Check whether the Logtail installation parameter and installation method are correctly configured.
- If Logtail is not installed, install Logtail by following the instructions that are provided in Install Logtail on a Linux server or Install Logtail on a Windows server.
  Important
  When you install Logtail, you must select an operating system that supports Logtail, select an installation parameter based on the region where your Log Service project resides, and select an installation method based on the network type that is used for log collection. For more information about network types, see Select a network type.

Step 2: Check whether the Logtail installation parameter and installation method are correctly configured

When you install Logtail, you must specify a correct Log Service endpoint for Logtail to connect to Log Service. You must select a Logtail installation parameter based on the region where your Log Service project resides and select an installation method based on the network type that is used for log collection. For more information, see Region names for Logtail installation and Select a network type. If the installation parameter is incorrectly configured or the Logtail installation script is invalid, heartbeats may not be detected on the server on which Logtail is installed. For more information about Log Service endpoints for different regions, see Endpoints.

The Logtail configuration file ilogtail_config.json contains the Logtail installation parameter and installation method. The following list describes the paths to the file:

Linux: /usr/local/ilogtail/ilogtail_config.json
32-bit Windows: C:\Program Files\Alibaba\Logtail\ilogtail_config.json
64-bit Windows: C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json

You can perform the following steps to check the Logtail installation parameter and installation method:

Check whether the region of the Log Service endpoint that is used by Logtail is the same as the region of your project in the ilogtail_config.json file.
1. Run the following command on your server to view the region of the Log Service endpoint that is used by Logtail:
```
cat /usr/local/ilogtail/ilogtail_config.json
```
  In this example, the following output is returned, which indicates that Logtail is installed on an Elastic Compute Service (ECS) instance in the China (Hangzhou) region.
2. View the region of your project in the Log Service console.
  If the region of the project is the same as the region that is recorded in the config_server_address field, the installation parameter is correctly configured.
View the endpoint that is specified in the ilogtail_config.json file and check whether the selected installation method is correct based on the network type of your server.
For example, the endpoint cn-hangzhou-intranet.log.aliyuncs.com is specified in the ilogtail_config.json file.
- Linux:
  Run the following command to test the network connectivity:
  curl logtail.cn-hangzhou-intranet.log.aliyuncs.com
  If an output similar to the following example is returned, the network is connected:
```
{"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"5DD39230BE9910FC6CF17609"}}
```
- Windows:
  Run the following command to test the network connectivity:
  telnet logtail.cn-hangzhou-intranet.log.aliyuncs.com 80
  If an output similar to the following example is returned, the network is connected:
```
Trying 100*0*7*5...
Connected to logtail.cn-hangzhou-intranet.log.aliyuncs.com.
Escape character is '^]'. 
```
Check whether the Logtail installation parameter and installation method are correctly configured.
- If Logtail is not installed, select an operating system that supports Logtail, select an installation parameter based on the region where your Log Service project resides, and select an installation method based on the network type of your server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.
- If Logtail is installed, perform the steps that are described in Step 3: Check whether the IP address that is specified in the machine group is correct.

Step 3: Check whether the IP address that is specified in the machine group is correct

The server IP address that is obtained by Logtail must be the same as the IP address that is specified in the machine group. Otherwise, the machine group has no heartbeats, or logs cannot be collected. Logtail obtains a server IP address by using the following methods:

If the hostname of the server is not bound to an IP address, Logtail obtains the IP address of the first network interface controller (NIC) of the server.
If the hostname of the server is bound to an IP address, Logtail obtains the IP address. You can view the hostname and IP address in the /etc/hosts file.
Note
You can obtain a hostname from the hostname field.

You can perform the following steps to check whether the IP address that is specified in the machine group is correct:

View the server IP address that is obtained by Logtail.
The ip field in the app_info.json file indicates the server IP address that is obtained by Logtail. The file is stored in one of the following paths:
- Linux: /usr/local/ilogtail/app_info.json
- 32-bit Windows: C:\Program Files\Alibaba\Logtail\app_info.json
- 64-bit Windows: C:\Program Files (x86)\Alibaba\Logtail\app_info.json
Important
If the ip field in the app_info.json file is empty, Logtail cannot work. You can specify the server IP address by using the Logtail startup parameter working_ip and then restart Logtail. For more information, see Configure the startup parameters of Logtail.
The app_info.json file is used only to record information. If you specify the IP address in the file, the server IP address that is obtained by Logtail is not updated.
View the IP address that is specified in the machine group.
For more information, see Manage machine groups.
Check whether the IP address that is obtained by Logtail client is same as the IP address that is specified in the machine group.
- If the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail, perform the steps that are described in Step 4: Check whether a user identifier is configured.
- If the IP address that is specified in the machine group is different from the IP address that is obtained by Logtail, change the IP address in the machine group. For more information, see Modify a machine group.
  - If the IP address that is specified in the machine group is incorrect, change the IP address in the machine group. Wait 1 minute and then check the heartbeat status of the machine group.
  - If you change the network configurations of the server on which Logtail is installed, restart Logtail to update the IP address that is obtained by Logtail. For example, you can modify the /etc/hosts file to change the network configurations. Then, change the IP address in the machine groups to the value of the ip field in the app_info.json file.
    You can restart Logtail by using the following methods:
    Linux:
    sudo /etc/init.d/ilogtaild stop sudo /etc/init.d/ilogtaild start
    Windows:
    Open the Run window and enter services.msc to open the Services window.
    Restart the LogtailDaemon service (Logtail V1.0.0.0 or later) or LogtailWorker service (Logtail V0.x.x.x).

Step 4: Check whether a user identifier is configured

If your server is an ECS instance that belongs to another Alibaba Cloud account, a server that is provided by a third-party cloud service provider, or a self-managed data center, you must specify the ID of the Alibaba Cloud account for which Log Service is activated as a user identifier for your server after you install Logtail on your server. This way, Logtail is authorized to collect logs from your server across different accounts. For more information, see Configure a user identifier.

Important

The user identifier must be the ID of an Alibaba Cloud account. For more information, see Obtain the ID of the Alibaba Cloud account for which Log Service is activated.

Linux:
Check whether a file named after the ID of the Alibaba Cloud account exists in the /etc/ilogtail/users directory.
- If yes, the user identifier is configured.
- If no, configure a user identifier. For more information, see Configure a user identifier.
Windows:
Check whether a file named after the ID of the Alibaba Cloud account exists in the C:\LogtailData\users directory.
- If yes, the user identifier is configured.
- If no, configure a user identifier. For more information, see Configure a user identifier.