All Products
Search
Document Center

Log Service:How do I troubleshoot the errors that are related to Logtail machine groups?

Last Updated:May 19, 2023

This topic describes how to troubleshoot the issue that a Logtail machine group does not have heartbeats.

Step 1: Check whether Logtail runs as expected

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the following command:

    ps -ef | grep ilogtail
    • If the output contains information that is similar to the following records, Logtail runs as expected. The sample records indicate the Logtail daemon process and the Logtail worker process.

      UID          PID    PPID  C STIME TTY          TIME CMD
      ...
      root          12       1  0 Nov10 ?        00:00:00 /usr/local/ilogtail/ilogtail
      root          14      12  0 Nov10 ?        03:07:43 /usr/local/ilogtail/ilogtail
      ...
      Important

      If three or more Logtail running records are returned, multiple Logtail instances are running on the current server. In this case, duplicate data may be collected. You must check whether the result meets your expectations.

    • If the output indicates that the Logtail-related processes did not run as expected, install Logtail. For more information, see Install Logtail on a Linux server.

      Important

      When you install Logtail, you must select an operating system that supports Logtail, configure the required installation parameter based on the region where your Log Service project resides, and then select an installation method based on the network type that is used for log collection. For information about network types, see Select a network type.

Windows

  1. Log on to the server on which Logtail is installed.

  2. Open the Run command window and enter services.msc.

  3. View the status of the LogtailDaemon service (Logtail V1.0.0.0 or later) or LogtailWorker service (Logtail V0.x.x.x).

    If the Logtail-related service is not running, install Logtail. For more information, see Install Logtail on a Windows server.

    Important

    When you install Logtail, you must select an operating system that supports Logtail, configure the required installation parameter based on the region where your Log Service project resides, and select an installation method based on the network type that is used for log collection. For information about network types, see Select a network type.

Step 2: Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail

Logtail obtains a server IP address by using the following methods:

  • If the hostname of the server is not bound to an IP address, Logtail obtains the IP address of the first network interface controller (NIC) of the server.

  • If you bind the hostname of the server to an IP address in the /etc/hosts file, Logtail obtains the IP address.

To check whether the IP address that is specified in the machine group is correct, perform the following steps:

  1. View the app_info.json file.

    Logtail records the obtained IP address in the ip field of the app_info.json file. The default path of the file varies based on the operating system.

    • Linux: /usr/local/ilogtail/app_info.json

    • 64-bit Windows: C:\Program Files (x86)\Alibaba\Logtail\app_info.json

    • 32-bit Windows: C:\Program Files\Alibaba\Logtail\app_info.json

      {
        "UUID" : "",
        "hostname" : "iZ8vbdlzf******azuhZ",
        "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
        "ip" : "172.**.**.200",
        "logtail_version" : "1.3.1",
        "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
        "update_time" : "2022-12-27 05:38:33"
      }
  2. Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail.

    Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Introduction.

    • If you use an IP address-based machine group, view the IP address in the IP Address field. For more information, see View the status of a machine group.

      If the value of the IP Address parameter is another IP address of Logtail, such as a public endpoint, change the value to the IP address that is obtained by Logtail, and then check whether the heartbeats of the machine can be detected. If the heartbeats can be detected, you can stop the troubleshooting process.

    • If you use a custom identifier-based machine group, use the IP address that is obtained by Logtail to query the status of the machine group. For more information, see View the status of a machine group. If the status is queried, you can stop the troubleshooting process.image..png

Step 3: Check whether the Logtail startup parameters are invalid

The ilogtail_config.json file records the startup parameters of Logtail.

  1. Log on to the server on which Logtail is installed.

  2. Find the ilogtail_config.json file.

    Run the following command to check whether the storage path of the file is specified in the environment variable:

    echo $ALIYUN_LOGTAIL_CONFIG
    • If the output is not empty, the value of the returned environment variable is the storage path of the ilogtail_config.json file. In most cases, the storage path is in the /etc/ilogtail/conf/<Project region>/ilogtail_config.json format.

    • If the output is empty, the storage path of the ilogtail_config.json file is not specified by the environment variable. You can use one of the following default paths for the file:

      • Linux: /usr/local/ilogtail/ilogtail_config.json

      • 64-bit Windows: C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json

      • 32-bit Windows: C:\Program Files\Alibaba\Logtail\ilogtail_config.json

  3. Open the ilogtail_config.json file and check whether the parameters in the configuration file are valid.

    {
      "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
      "data_server_list" :
      [
        {
          "cluster" : "<Project region>",
          "endpoint" : "<endpoint>"
        }
      ],
      ...
    }
    • If the startup parameters in the ilogtail_config.json file are configured based on the descriptions in the following table, the startup parameters of Logtail are valid.

    • If the startup parameters of Logtail are invalid, modify the ilogtail_config.json file based on the descriptions in the following table and restart Logtail. For more information, see Appendix: Restart Logtail.

      Scenario

      Network type

      <config_region>

      <endpoint>

      The server is an Elastic Compute Service (ECS) instance that resides in the same region as the project.

      Alibaba Cloud internal network

      <Project region>-intranet

      <Project region>-intranet.log.aliyuncs.com

      Others

      Internet

      <Project region>

      <Project region>.log.aliyuncs.com

      Global acceleration

      log-global.aliyuncs.com

Step 4: Check whether the network connection is established

If data is uploaded by using Logtail, make sure that the server on which Logtail is installed can connect to the following addresses.

Important

If you use an internal network, you must add -intranet to the address.

  1. The address and HTTPS version that are specified by the config_server_address field in the ilogtail_config.json file.

  2. http://<Project name>.<endpoint>. <endpoint> indicates the address that is specified by the data_server_list.endpoint field in the ilogtail_config.json file.

  3. http://ali-<Project region>-sls-admin.<endpoint>. <endpoint> indicates the address that is specified by the data_server_list.endpoint field in the ilogtail_config.json file.

To check whether the network connection is established, perform the following steps:

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the curl command to connect to the preceding addresses in sequence.

    curl xxx

    If each output contains the information that is similar to the following code, the network connection is established.

    {"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"5D****09"}}

    If the network connection is not established, check whether port 80 and port 443 in the network environment are enabled, check whether the destination address is blocked, and check whether other network-side configurations are valid, such as DNS configurations and security groups.

Windows

  1. Log on to the server on which Logtail is installed.

  2. Run the telnet command to connect to the preceding addresses in sequence.

    telnet xxx 80 # If the HTTPS protocol is used, the port number is 443.

    If each output contains the information that is similar to the following code, the network connection is established.

    Trying 100*0*7*5...
    Connected to xxx.
    Escape character is '^]'.

    If the network connection is not established, check whether port 80 and port 443 in the network environment are enabled, check whether the destination address is blocked, and check other network-side configurations are valid, such as DNS configurations and security groups.

Step 5: Check whether the system time of the server on which Logtail is installed is correct

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the date command to view the system time.

    Wed Dec 28 06:59:26 UTC 2022

    If the system time is faster or slower than the local time, perform the following operations:

    • Change the system time to the local time.

    • If the system time cannot be changed, modify the startup parameters of Logtail by adding the following parameter setting to the ilogtail_config.json file: "enable_log_time_auto_adjust": true. After you modify the startup parameters, restart Logtail. For more information, see Appendix: Restart Logtail. For information about the path of the ilogtail_config.json file, see Step 3: Check whether the startup parameters of Logtail are valid.

Windows

  1. Log on to the server on which Logtail is installed.

  2. View the time information in the taskbar in the lower-right corner of the desktop.

    • Change the system time to the local time.

    • If the system time cannot be changed, modify the startup parameters of Logtail by adding the following parameter setting to the ilogtail_config.json file: "enable_log_time_auto_adjust": true. After you modify the startup parameters, restart Logtail. For more information, see Appendix: Restart Logtail. For information about the path of the ilogtail_config.json file, see Step 3: Check whether the startup parameters of Logtail are valid.

Step 6: Check whether a user identifier is configured

If your server is an ECS instance that belongs to another Alibaba Cloud account, a server provided by a third-party cloud service provider, or a server deployed in a self-managed data center, you must authorize Logtail to collect logs from the server across accounts. After you install Logtail on your server, you must specify the ID of the Alibaba Cloud account for which Log Service is activated as a user identifier. For more information, see Configure a user identifier.

Important

The user identifier must be the ID of an Alibaba Cloud account. For more information, see Obtain the ID of the Alibaba Cloud account for which Log Service is activated.

You can check whether a user identifier is configured by using the ALIYUN_LOGTAIL_USER_ID environment variable or the user identifier file in the specified directory. If a user identifier is configured by using the environment variable, the configuration of the environment variable take precedence. Run the following command to check whether the user identifier is specified by using the environment variable.

echo $ALIYUN_LOGTAIL_USER_ID
  • If the output contains the ID of the Alibaba Cloud account to which the current project belongs, the user identifier is valid.

  • If the output is not empty but does not contain the ID of the Alibaba Cloud account to which the project belongs, you must set the value of the environment variable to the ID of the Alibaba Cloud account to which the project belongs or add the ID of the Alibaba Cloud account to which the project belongs. Separate multiple IDs with commas (,).

  • If the output is empty, you must check whether a user identifier file exists in the specified path.

    Note

    The path of the user identifier file varies based on the operating system.

    • Linux: /etc/ilogtail/users/

    • Windows: C:\LogtailData\users\

    • If no user identifier file exists in the specified path or the user identifier file is invalid, perform the following steps to fix the issue:

      • Linux: Run the cd /etc/ilogtail/users/ && touch <uid> command to create a user identifier file. <uid> specifies the ID of the Alibaba Cloud account to which the project belongs.

      • Windows: Go to the C:\LogtailData\users\ directory and create an empty file named <uid>. <uid> specifies the ID of the Alibaba Cloud account to which the project belongs.

    • If the specified path contains a file named after the ID of the Alibaba Cloud account to which the current project belongs, the user identifier is valid.

Important

After you perform the preceding operations, you must restart Logtail. For more information, see Appendix: Restart Logtail.

Step 7: Check whether a custom identifier is specified if you use a custom identifier-based machine group

Check whether a custom identifier is specified by using the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable or the user_defined_id file in the specified directory. If a custom identifier is specified by using the environment variable, the configuration of the environment variable take precedence. Run the following command to check whether the custom identifier is specified by using the environment variable.

echo $ALIYUN_LOGTAIL_USER_DEFINED_ID
  • If the output contains the custom identifier that you specified for the machine group, the custom identifier is valid.

  • If the output is not empty but does not contain the custom identifier that you specified for the machine group, you must change the value of the environment variable to the custom identifier that you specified for the machine group or add the custom identifier of the machine group. Separate multiple custom identifiers with commas (,).

  • If the output is empty, you must check whether the user_defined_id file exists or whether the custom identifier is specified in the file.

    Note

    The path of the user_defined_id file varies based on the operating system.

    • Linux: /etc/ilogtail/user_defined_id

    • Windows: C:\LogtailData\user_defined_id/

    • If the user_defined_id file does not exist, add a user_defined_id file and enter the custom identifier of the machine group in the file.

    • If no custom identifier exists in the user_defined_id file or the custom identifier is invalid, add a line in the file and enter the custom identifier of the machine group.

    • If the user_defined_id file contains the custom identifier that you specified for the machine group, the custom identifier is valid.

Important

After you perform the preceding operations, you must restart Logtail. For more information, see Appendix: Restart Logtail.

What to do next

If the issue persists after you perform the preceding troubleshooting operations, submit a ticket.

Appendix: Restart Logtail

Linux

  1. Log on to the server on which Logtail is installed.

  2. Run the following command:

    sudo /etc/init.d/ilogtaild restart

Windows

  1. Log on to the server on which Logtail is installed.

  2. Open the Run command window and enter services.msc.

  3. Restart the LogtailDaemon service (Logtail V1.0.0.0 or later) or LogtailWorker service (Logtail V0.x.x.x).