All Products
Search
Document Center

Simple Log Service:How do I troubleshoot an error that is related to a Logtail machine group in a host environment?

Last Updated:Mar 04, 2024

This topic describes how to troubleshoot the error that a Logtail machine group has no heartbeats in a host environment.

Step 1: Check whether Logtail runs as expected

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the following command:

    ps -ef | grep ilogtail
    • If the following two records are returned, Logtail runs as expected. The records indicate the Logtail daemon process and the Logtail worker process.

      UID          PID    PPID  C STIME TTY          TIME CMD
      ...
      root          12       1  0 Nov10 ?        00:00:00 /usr/local/ilogtail/ilogtail
      root          14      12  0 Nov10 ?        03:07:43 /usr/local/ilogtail/ilogtail
      ...
      Important

      If three or more Logtail running records are returned, multiple Logtail processes are running on the current server. In this case, duplicate data may be collected. You must check whether the running Logtail processes meet your business requirements.

    • If the output indicates that no Logtail processes run, install Logtail. For more information, see Install Logtail on a Linux server.

      Important

      When you install Logtail on a server, make sure that the operating system of the server supports Logtail. You can select installation parameters based on the region of your Simple Log Service project and select an installation method based on the network type that you use. For more information about network types, see Select a network type.

Windows

  1. Log on to the server on which Logtail is installed.

  2. Open the Run window and enter services.msc to open the Services window.

  3. If you use Logtail V1.0.0.0 or later, view the status of the LogtailDaemon service. If you use Logtail V0.x.x.x, view the status of the LogtailWorker service.

    If the services are not in the Running state, install Logtail. For more information, see Install Logtail on a Windows server.

    Important

    When you install Logtail on a server, make sure that the operating system of the server supports Logtail. You can select installation parameters based on the region of your Simple Log Service project and select an installation method based on the network type that you use. For more information about network types, see Select a network.

Step 2: Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail

Logtail obtains a server IP address by using the following methods:

  • If the hostname of the server is not mapped to an IP address, Logtail obtains the IP address of the first network interface controller (NIC) of the server.

  • If the hostname of the server is mapped to an IP address, Logtail obtains the IP address. To map a hostname to an IP address, you can edit the /etc/hosts file.

To check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail, perform the following operations:

  1. Open the app_info.json file.

    The value of the ip field specifies the IP address that is obtained by Logtail. The following table describes the default paths to the app_info.json file in different operating systems.

    Operating system

    Logtail

    Path to the app_info.json file

    Linux

    Logtail (64-bit)

    /usr/local/ilogtail/app_info.json

    64-bit Windows

    Logtail (64-bit)

    C:\Program Files\Alibaba\Logtail\app_info.json

    Logtail (32-bit)

    C:\Program Files (x86)\Alibaba\Logtail\app_info.json

    32-bit Windows

    Logtail (32-bit)

    C:\Program Files\Alibaba\Logtail\app_info.json

  2. Obtain the value of the ip field in the app_info.json file.

    {
      "UUID" : "",
      "hostname" : "iZ8vbdlzf******azuhZ",
      "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
      "ip" : "172.**.**.200",
      "logtail_version" : "1.3.1",
      "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
      "update_time" : "2022-12-27 05:38:33"
    }
  3. Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail.

    Simple Log Service machine groups are classified into IP address-based machine groups and custom identifier-based machine groups. For more information, see Introduction.

    • If you use an IP address-based machine group, view the IP address in the IP Addresses field. For more information, see View the status of a machine group.

      If the value of the IP Addresses field is another IP address of Logtail, such as a public IP address, change the value to the IP address that is obtained by Logtail. Then, check whether the heartbeat of the machine group can be detected. If the heartbeat can be detected, you can stop troubleshooting.

    • If you use a custom identifier-based machine group, use the IP address that is obtained by Logtail to query the status of the machine group. For more information, see View the status of a machine group. If the status is queried, you can stop troubleshooting.image..png

Step 3: Check whether the Logtail startup parameters are valid

The ilogtail_config.json file records the startup parameters of Logtail.

  1. Log on to the server on which Logtail is installed.

  2. Find the ilogtail_config.json file.

    Run the following command to check whether the file path can be obtained from the environment variable:

    echo $ALIYUN_LOGTAIL_CONFIG
    • If the output is not empty, the return value of the environment variable is the path to the ilogtail_config.json file. In most cases, the path is in the /etc/ilogtail/conf/<Region to which the project belongs>/ilogtail_config.json format.

    • If the output is empty, the path to the ilogtail_config.json file cannot be obtained from the environment variable. You can use one of the following default paths to find the ilogtail_config.json file.

      Operating system

      Logtail

      Path to the ilogtail_config.json file

      Linux

      Logtail (64-bit)

      /usr/local/ilogtail/ilogtail_config.json

      64-bit Windows

      Logtail (64-bit)

      C:\Program Files\Alibaba\Logtail\ilogtail_config.json

      Logtail (32-bit)

      C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json

      32-bit Windows

      Logtail (32-bit)

      C:\Program Files\Alibaba\Logtail\ilogtail_config.json

  3. Open the ilogtail_config.json file and check whether the parameters in the configuration file are valid.

    {
      "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
      "data_server_list" :
      [
        {
          "cluster" : "<Region to which the project belongs>",
          "endpoint" : "<endpoint>"
        }
      ],
      ...
    }
    • If the startup parameters in the ilogtail_config.json file are configured based on the descriptions in the following table, the startup parameters of Logtail are valid.

    • If the startup parameters of Logtail are invalid, modify the ilogtail_config.json file based on the descriptions in the following table and restart Logtail. For more information, see Appendix: Restart Logtail.

      Scenario

      Network type

      <config_region>

      <endpoint>

      The server is an Elastic Compute Service (ECS) instance that resides in the same region as the project.

      Alibaba Cloud internal network

      <Region to which the project belongs>-intranet

      <Region to which the project belongs>-intranet.log.aliyuncs.com

      Other scenarios

      Internet

      <Region to which the project belongs>

      <Region to which the project belongs>.log.aliyuncs.com

      Global acceleration

      log-global.aliyuncs.com

Step 4: Check whether network connectivity is available

Logtail can upload data to Simple Log Service only if the server on which Logtail runs can connect to the following addresses.

Important

If you use an internal network, you must add -intranet to the address.

  1. The address and HTTPS version that are specified in the value of the config_server_address field in the ilogtail_config.json file.

  2. http://<Project name>.<endpoint>. <endpoint> specifies the address that is specified by the data_server_list.endpoint field in the ilogtail_config.json file.

  3. http://ali-<Region to which the project belongs>-sls-admin.<endpoint>. <endpoint> specifies the address that is specified by the data_server_list.endpoint field in the ilogtail_config.json file.

To check whether network connectivity is available, perform the following operations:

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the curl command to connect to the preceding addresses in sequence.

    curl xxx

    If each output contains the information that is similar to the following code, network connectivity is available.

    {"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"5D****09"}}

    If network connectivity is unavailable, check whether port 80 and port 443 in the network environment are enabled, whether the destination address is blocked, and whether other network-side configurations such as DNS configurations and security groups are valid.

Windows

  1. Log on to the server on which Logtail is installed.

  2. Run the telnet command to connect to the preceding addresses in sequence.

    telnet xxx 80 # If HTTPS is used, specify port 443.

    If each output contains the information that is similar to the following code, network connectivity is available.

    Trying 100*0*7*5...
    Connected to xxx.
    Escape character is '^]'.

    If network connectivity is unavailable, check whether port 80 and port 443 in the network environment are enabled, whether the destination address is blocked, and whether other network-side configurations such as DNS configurations and security groups are valid.

Step 5: Check whether the system time of the server on which Logtail is installed is correct

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the date command to view the system time.

    Wed Dec 28 06:59:26 UTC 2022

    If the system time is earlier or later than the local time, perform the following operations:

Windows

  1. Log on to the server on which Logtail is installed.

  2. View the time information in the taskbar in the lower-right corner of the desktop.

Step 6: Check whether a user identifier is configured

If you use an ECS instance that belongs to a different Alibaba Cloud account from your Simple Log Service, a server from a third-party cloud service provider, or a server in a data center and you want to use Logtail to collect logs from the server, you must specify the ID of the Alibaba Cloud account for which Simple Log Service is activated as a user identifier after you install Logtail on the server. This way, the account is granted the required permissions to use Logtail to collect logs from the server. For more information, see Configure a user identifier.

Important

The user identifier must be the ID of an Alibaba Cloud account. For more information, see Obtain the ID of the Alibaba Cloud account for which Simple Log Service is activated.

You can check whether a user identifier is configured by using the ALIYUN_LOGTAIL_USER_ID environment variable or a user identifier file in a specified directory. If a user identifier is configured by using the environment variable, the configuration of the environment variable takes precedence. You can run the following command to check whether the user identifier is configured by using the environment variable:

echo $ALIYUN_LOGTAIL_USER_ID
  • If the output contains the ID of the Alibaba Cloud account to which the project belongs, the user identifier is configured and valid.

  • If the output is not empty but does not contain the ID of the Alibaba Cloud account to which the project belongs, you must change the value of the environment variable to the ID of the Alibaba Cloud account to which the project belongs or append the ID to the existing value. Separate multiple IDs with commas (,).

  • If the output is empty, you must check whether a user identifier file exists in a specified directory.

    Note

    The path to a user identifier file varies based on the operating system.

    • Linux: /etc/ilogtail/users/

    • Windows: C:\LogtailData\users\

    • If no user identifier file exists in a specified directory or a user identifier file is invalid, perform the following operations to fix the issue:

      • Linux: Run the cd /etc/ilogtail/users/ && touch <uid> command to create a user identifier file. <uid> specifies the ID of the Alibaba Cloud account to which the project belongs.

      • Windows: Go to the C:\LogtailData\users\ directory and create an empty file named <uid>. <uid> specifies the ID of the Alibaba Cloud account to which the project belongs.

    • If the specified directory contains a file named after the ID of the Alibaba Cloud account to which the project belongs, the user identifier is configured and valid.

Important

After you perform the preceding operations, you must restart Logtail. For more information, see Appendix: Restart Logtail.

Step 7: Check whether a custom identifier is configured if you use a custom identifier-based machine group

You can check whether a custom identifier is configured for your server by using the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable or the user_defined_id file in a specified directory. If a custom identifier is configured by using the environment variable, the configuration of the environment variable takes precedence. You can run the following command to check whether the custom identifier is configured by using the environment variable:

echo $ALIYUN_LOGTAIL_USER_DEFINED_ID
  • If the output contains the custom identifier that you specified for the machine group, the custom identifier is configured and valid.

  • If the output is not empty but does not contain the custom identifier that you configured for the machine group, you must change the value of the environment variable to the custom identifier that you configured for the machine group or append the custom identifier to the existing value. Separate multiple custom identifiers with commas (,).

  • If the output is empty, you must check whether the user_defined_id file exists or whether a custom identifier is configured in the file.

    Note

    The path to the user_defined_id file varies based on the operating system.

    • Linux: /etc/ilogtail/user_defined_id

    • Windows: C:\LogtailData\user_defined_id

    • If the user_defined_id file does not exist, create a file named user_defined_id and add the custom identifier of the machine group to the file. For more information, see Create a custom identifier-based machine group.

    • If no custom identifier exists in the user_defined_id file or the custom identifier is invalid, add a line to the file and enter the custom identifier of the machine group. For more information, see Create a custom identifier-based machine group.

    • If the user_defined_id file contains the custom identifier that you configured for the machine group, the custom identifier is configured and valid.

Important

After you perform the preceding operations, you must restart Logtail. For more information, see Appendix: Restart Logtail.

What to do next

If the error persists after troubleshooting, submit a ticket.

Appendix: Restart Logtail

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the following command:

    sudo /etc/init.d/ilogtaild restart

Windows

  1. Log on to the server on which Logtail is installed.

  2. Open the Run window and enter services.msc to open the Services window.

  3. If you use Logtail V1.0.0.0 or later, restart the LogtailDaemon service. If you use Logtail V0.x.x.x, restart the LogtailWorker service.