Simple Log Service:Troubleshooting Logtail machine group issues (host scenario)

Machine group installation examples

Troubleshooting checklist

1. Step 1: Verify that Logtail runs on the server.

2. Step 2: Confirm that the IP address in the machine group matches the IP address obtained by Logtail. A mismatch causes heartbeat failures.

3. Step 3: Check whether the project region configured in the ilogtail_config.json file is correct.

4. Step 4: Verify network connectivity between the server and the project.

5. Step 5: Check whether the system time on the Logtail server is correct. If the system time deviates significantly from the actual time, update it.

6. Step 6: For cross-account collection, if the server is not an ECS instance or the ECS instance and the project belong to different Alibaba Cloud accounts, verify that the user identifier is configured.

7. Step 7: For custom identifier-based machine groups, if the machine group uses a custom identifier, verify that the custom identifier is configured on the server.

8. Step 8: After completing the preceding changes, restart Logtail.

What to do next

If logs are still not collected after the heartbeat status becomes OK, troubleshoot based on Logtail error messages. For more information, see How to view Logtail collection errors.

Step 1: Verify that Logtail runs as expected

1. Obtain the value of the ip field from the app_info.json file.

   The default paths to this file on different systems are listed in the following table:

   Operating system	Logtail	Path to app_info.json
   Linux	Logtail (64-bit)	/usr/local/ilogtail/app_info.json
   Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\app_info.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json
   Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\app_info.json

   Logtail records the obtained IP address in the ip field of the app_info.json file.

   {
     "UUID" : "",
     "hostname" : "iZ8vbdlzf******azuhZ",
     "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
     "ip" : "172.**.**.200",
     "logtail_version" : "1.3.1",
     "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
     "update_time" : "2022-12-27 05:38:33"
   }

2. Verify that the machine group uses the IP address that Logtail obtains.

   Simple Log Service machine groups include IP address-based machine groups and custom identifier-based machine groups. For more information, see Machine groups.

   • IP address-based machine group: Verify that the IP Address field contains the IP address from the previous step.

     If the IP address does not match, confirm the correct IP address of the host. If the IP Address text box contains a different IP address for the host, such as a public IP address, update the IP address in the machine group. If the IP address obtained in the previous step is incorrect, modify the working_ip parameter in Logtail startup parameters and restart Logtail. Then, monitor the machine heartbeat. If the heartbeat recovers, troubleshooting is complete.

   • Custom identifier-based machine group: Check whether the Machine Group Status includes the IP address obtained in the previous step. If the heartbeat status shows OK, troubleshooting is complete.

The ilogtail_config.json file stores Logtail's startup parameters.

1. Log on to the server where Logtail is installed.

2. Locate the ilogtail_config.json file.

   The default paths to this file on different systems are listed in the following table:

   Operating system	Logtail	Path to ilogtail_config.json
   Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail_config.json
   Windows (64-bit)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json
   Windows (32-bit)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

   1. Open the ilogtail_config.json file and verify that the configuration parameters are correct.

      {
        "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
        "data_server_list" :
        [
          {
            "cluster" : "<project region>",
            "endpoint" : "<endpoint>"
          }
        ],
        ...
      }

      • If the startup parameters in the ilogtail_config.json file match the descriptions in the following table, the Logtail startup parameters are correct.

      • If the Logtail startup parameters are incorrect, update the ilogtail_config.json file based on the following table and restart Logtail. For details, see Restart Logtail.

        For project region information, see Supported regions.

        Scenario	Network type	<config_region>	<endpoint>
        Server is an ECS instance in the same region as the project	Alibaba Cloud internal network	<project region>-intranet	<project region>-intranet.log.aliyuncs.com
        Other scenarios	Internet	<project region>	<project region>.log.aliyuncs.com
        	Transfer acceleration		log-global.aliyuncs.com

For Logtail to upload data, the server must be able to connect to the following addresses.

1. The address specified by the config_server_address field in the ilogtail_config.json file and its HTTPS version.

2. http://<project name>.<endpoint>.

   • View the project name and region as shown below.

     

   • <endpoint> is the address specified by the data_server_list.endpoint field in the ilogtail_config.json file.

3. http://ali-<project region>-sls-admin.<endpoint>. Here, <endpoint> is the address specified by the data_server_list.endpoint field in the ilogtail_config.json file.

Perform the following checks and solutions:

Linux

1. Log on to the server where Logtail is installed.

2. Run the date command to view the system time.

   Wed Dec 28 06:59:26 UTC 2022

   If the system time deviates significantly from the actual time, take one of the following actions.

   • Adjust the system time to the actual time.

   • If you cannot change the system time, add the configuration item "enable_log_time_auto_adjust": true to the ilogtail_config.json file. After modifying the file, restart Logtail. For details, see Restart Logtail. For the ilogtail_config.json file path, see Step 3: Verify Logtail startup parameters.

Windows

1. Log on to the server where Logtail is installed.

2. Check the time in the taskbar in the lower-right corner of the desktop.

   • Adjust the system time to the actual time.

   • If you cannot change the system time, add the configuration item "enable_log_time_auto_adjust": true to the ilogtail_config.json file. After modifying the file, restart Logtail. For details, see Restart Logtail. For the ilogtail_config.json file path, see Step 3: Verify Logtail startup parameters.

Step 6: Verify the user identifier for cross-account collection

Check the user identifier file in the specified directory. If the result is empty, verify whether the user identifier file exists in the specified path. The user identifier confirms that the account has permission to access the server.

• If no user identifier file exists in the specified path or the file is misconfigured, resolve the issue as follows.

  • Linux: Run the command cd /etc/ilogtail/users/ && touch <uid> to create a user identifier file. Replace <uid> with the Alibaba Cloud account ID of the project.

  • Windows: Go to the C:\LogtailData\users\ directory and create an empty file named <uid>. Replace <uid> with the Alibaba Cloud account ID of the project.

• If a file named with the Alibaba Cloud account ID of the current project exists in the specified path, the user identifier is correctly configured.

Step 7: Verify the custom identifier for custom identifier-based machine groups

If you use a custom identifier-based machine group, check the user_defined_id file in the specified directory to verify whether the custom identifier is configured on the server.

• If the result is empty, check whether the user_defined_id file exists and whether it contains the custom identifier.

  Note

  Paths to the user_defined_id file:

  • Linux: /etc/ilogtail/user_defined_id

  • Windows: C:\LogtailData\user_defined_id

  • If the user_defined_id file does not exist, create a file named user_defined_id and enter the custom identifier of the machine group. For details, see Configure a custom identifier.

  • If the user_defined_id file does not contain the custom identifier or the identifier is incorrect, add a new line and enter the custom identifier of the machine group. For details, see Configure a custom identifier.

  • If the user_defined_id file already contains the custom identifier you set for the machine group, the custom identifier is correctly configured.

Step 8: Restart Logtail

After you complete the preceding changes, restart Logtail.