Troubleshoot heartbeat issues

Causes of heartbeat issues

LoongCollector uses the following configuration items to identify the destination project and report heartbeats. To resolve heartbeat issues, verify these items and check network connectivity.

• The Alibaba Cloud account that owns the SLS project: This account must have permissions to access and collect logs from the server.

• The region and connection type of the project: The SLS endpoint is dynamically generated based on the region and connection type. Ensure the server can connect to this endpoint. Network connection types and endpoints.

• Custom identifier or IP address: LoongCollector establishes a heartbeat by associating the server with a machine group through an IP address or custom identifier.

Heartbeat establishment process

1. LoongCollector reads its configuration to retrieve Alibaba Cloud account IDs, the endpoint (generated from the region and connection type), and the server's IP address or custom identifier.

2. LoongCollector reports a heartbeat to a project in the specified region.

3. Projects that meet the criteria compare the IP address or custom identifier in their machine groups with the reported information.

4. If the information matches, a heartbeat is established and the status in the corresponding machine group changes to OK.

Common scenarios for heartbeat issues

New server heartbeat is FAIL

The initial connection may take time to establish. Wait about two minutes and refresh the status. If the heartbeat is still FAIL, follow these steps:

1. Verify that you selected the correct LoongCollector installation scenario. If incorrect, uninstall and reinstall LoongCollector.

   Installation method	Applicable scenario
   Same account and region	Applies when the server is an ECS instance in the same account and region as the project.
   Same account but different regions	Applies when the server is an ECS instance in the same account but a different region from the project.
   Different accounts but same region	Applies when the server is an ECS instance in the same region as the project but under a different account.
   Other cloud or on-premises servers	Applies when the server is not an ECS instance (on-premises or another cloud provider). Also applies when the ECS instance and the project are in different accounts and different regions. Treat the server as on-premises in this case.

2. On the server, run sudo /etc/init.d/loongcollectord status to check LoongCollector status. If the output is loongcollector is running, it has started. Otherwise, start it:

   If you use Logtail, run sudo /etc/init.d/ilogtaild status to check its status and sudo /etc/init.d/ilogtaild start to start it.

   sudo /etc/init.d/loongcollectord start

3. In a cross-account scenario, you must configure a user ID file to grant the project's account permissions to access the server and collect logs.

   Check the user ID file

   1. Check that the /etc/ilogtail/users/{Alibaba Cloud account ID} file exists. Create it if missing.

      1. Log in to the Simple Log Service console. Hover over the profile picture in the upper-right corner and copy the main account ID.

      2. On the server where LoongCollector is installed, create a user ID file. Use the main account ID as the filename.

         touch /etc/ilogtail/users/{Alibaba Cloud account ID} # Use the main account ID as the filename. No file extension is needed.

   2. Verify the filename meets these requirements:

      • The {Alibaba Cloud account ID} must be the ID of the main account.

      • The {Alibaba Cloud account ID} must be the ID of the Alibaba Cloud account that owns the SLS project, not the account that owns the server.

4. Verify the region and connection type, and confirm the server can reach the endpoint. Check if the region in /usr/local/ilogtail/ilogtail_config.json matches the Region ID of the SLS project. If not, modify the configuration:

   Test endpoint connectivity and modify the server configuration

   1. Log in to the Simple Log Service console. In the project list, click the destination project.

   2. Click the icon to the right of the project name to open the project overview page.

   3. In the Endpoint section, view the endpoint for the current project. Replace ${project_name} with your project name and ${endpoint} with the public endpoint, then run the following command on the server:

      curl https://${project_name}.${endpoint}

   4. A response like {"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"5D****09"}} indicates the network connection works. Otherwise, check if the destination is blocked and verify DNS, security group, and port settings (port 80 for HTTP, port 443 for HTTPS).

      This error is expected and confirms network connectivity. It occurs because the test command does not include parameters required for a valid API request.

   5. Modify the following parameters in the /usr/local/ilogtail/ilogtail_config.json file:

      • config_servers: The path for retrieving collection configurations. Set to "http://logtail.${endpoint}", where ${endpoint} is the public endpoint.

      • data_servers:

        • region: The region for data transmission. Set to "${region_id}", where ${region_id} is the Region ID of the SLS project.

        • endpoint_list: The path for data transmission. Set to "${endpoint}", where ${endpoint} is the public endpoint.

   6. Save the changes and restart LoongCollector.

      If you use Logtail, the restart command is: sudo /etc/init.d/ilogtaild restart

      sudo /etc/init.d/loongcollectord restart

5. Check the custom identifier or IP address:

   1. Log in to the Simple Log Service console. In the project list, click the destination project.

   2. In the left-side navigation pane, choose Resources > Machine Groups. On the Machine Groups page, click the target machine group.

   3. On the Machine Group Configurations page, check the Machine Group Identifier and perform the corresponding action:

      Custom identifier

      1. Check that /etc/ilogtail/user_defined_id exists on the server. Create it if missing.

      2. Write a custom string to this file as the custom identifier. This example uses user-defined-test-1.

         # Write a custom string to the specified file.
         echo "user-defined-test-1" > /etc/ilogtail/user_defined_id 

      3. In the console, set the value of Custom Identifier to the custom string. In this example, the value is user-defined-test-1.

      IP address

      Add the ip value from /usr/local/ilogtail/app_info.json on the server to the IP Address field in the console.

      IP address retrieval logic: If a hostname-to-IP-address mapping is configured in the /etc/hosts file on the server, the mapped IP address is used. If no mapping is configured, the IP address of the first network interface card (NIC) is used. If the working_ip parameter is set in the /usr/local/ilogtail/ilogtail_config.json file, its value is used as the server's IP address. Ensure that an IP address can be retrieved by using at least one of these methods. Otherwise, the ip field will be empty, and a heartbeat cannot be established.

Heartbeat changed from OK to FAIL

A previously successful heartbeat confirms your initial configuration was correct. If the machine group uses a custom identifier, the settings are static and the issue is likely network-related — check connectivity to the SLS endpoint. If the machine group uses an IP address, a FAIL status usually indicates an IP change or conflict. Follow these steps:

1. Restart LoongCollector on the server to retrieve the latest IP address.

   If you use Logtail, the restart command is: sudo /etc/init.d/ilogtaild restart

   sudo /etc/init.d/loongcollectord restart

2. View the ip field in /usr/local/ilogtail/app_info.json on the server.

   IP address retrieval logic: If a hostname-to-IP-address mapping is configured in the /etc/hosts file on the server, the mapped IP address is used. If no mapping is configured, the IP address of the first network interface card (NIC) is used. If the working_ip parameter is set in the /usr/local/ilogtail/ilogtail_config.json file, its value is used as the server's IP address.

3. Log in to the Simple Log Service console. In the project list, click the destination project.

4. In the left-side navigation pane, choose Resources > Machine Groups. On the Machine Groups page, click the target machine group.

5. On the Machine Group Configurations page, check whether the IP Address field contains the value of the ip field from the /usr/local/ilogtail/app_info.json file. If it does not, add the value of the ip field to the IP Address field.

6. If the IP address matches but the heartbeat remains FAIL, the IP may be unstable or conflicting. Consider switching to a custom identifier.

Heartbeat fails after switching identifier type

If IP address conflicts or changes make IP-based machine groups unsuitable, switch to a custom identifier. This change does not affect network connectivity, account information, or region settings. Ensure the custom identifier is configured correctly:

1. Confirm that /etc/ilogtail/user_defined_id exists. Create it if missing.

2. Write a custom string to this file as the custom identifier. This example uses user-defined-test-1.

   # Write a custom string to the specified file.
   echo "user-defined-test-1" > /etc/ilogtail/user_defined_id 

3. Log in to the Simple Log Service console. In the project list, click the destination project.

4. In the left-side navigation pane, choose Resources > Machine Groups. On the Machine Groups page, click the target machine group.

5. On the Machine Group Configurations page, confirm the following parameters. If incorrect, click Modify in the upper-right corner to update them.

   • Machine Group Identifier: Custom Identifier.

   • Custom identifier: The custom string. In this example, the value is user-defined-test-1.

FAQ

Why is heartbeat FAIL with correct configuration?

If configurations are correct and the network is working, the heartbeat may still be FAIL for these reasons:

• The heartbeat interval is throttled because no collection configuration has been applied in the region for a long time.

  • To reduce service load, the agent decreases its configuration request frequency when a region returns no configurations. The interval can increase up to 12 minutes. If this exceeds the heartbeat timeout threshold, the status changes to FAIL.

  • Solution: Ignore the FAIL status and apply a collection configuration to the machine group. The heartbeat recovers on the next request. To restore it immediately, restart the agent.

• The running configuration does not match the current ilogtail_config.json file.

  • Example: The agent starts with a non-default configuration, then ilogtail_config.json is modified without restarting the agent.

  • How to check:

    • The simplest solution is to restart the agent. It automatically loads the latest configuration.

    • Check the logs: If you want to avoid interrupting collection, check /usr/local/ilogtail/ilogtail.LOG. From the beginning of the file, search for load logtail config file. The corresponding entry shows the active configuration. Verify it matches your local file.