Simple Log Service:Troubleshoot container log collection issues

FAQ: Why is there no heartbeat after deploying Docker/LoongCollector?

If LoongCollector or Logtail shows no heartbeat or fails to register with the machine group after Docker deployment, the root cause is usually a missing user identifier (AliUID) configuration.

Solution:

1. On the host machine, create the /etc/ilogtail/users/ directory.

   mkdir -p /etc/ilogtail/users/

2. In the /etc/ilogtail/users/ directory, create an empty file named after your Alibaba Cloud account ID.

   touch /etc/ilogtail/users/<your-alibaba-cloud-account-id>

3. When starting the LoongCollector or Logtail container, mount the directory as read-only by adding the following flag:

   -v /etc/ilogtail/users:/etc/ilogtail/users:ro

4. Restart the LoongCollector or Logtail container.

Verification: Log on to the SLS console, go to Resources > Machine Groups, and open the target machine group. On the Machine Group Configurations page, confirm that the heartbeat status of the machine group has changed to OK.

FAQ: Why does the machine group show only one server when multiple Docker Swarm servers are reporting the same IP?

Cause: In a Docker Swarm cluster, multiple servers may report the same internal IP address through the container network. If those servers also share the same ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable value, the system cannot distinguish between them, and the Machine Groups displays only one entry.

Solution 1: Set a unique ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable for the Logtail container on each server.

Set a different value on each server and ensure the value matches the corresponding Machine Groups configuration. For example:

-e ALIYUN_LOGTAIL_USER_DEFINED_ID=<unique-id-for-this-server>

Solution 2: Set the ALIYUN_LOGTAIL_WORKING_IP environment variable to manually specify a unique IP address for each server.

Use a value that uniquely identifies each host, such as the host's public IP address or an internal network IP address that is unique across all servers:

-e ALIYUN_LOGTAIL_WORKING_IP=<unique-ip-for-this-server>

Machine group does not cover all container nodes

Symptom: The machine group heartbeat is normal, but log collection from some containers has stopped.

This issue may occur when the containers are running on servers that are not included in the Machine Groups.

Troubleshooting steps:

1. On all relevant servers, run the following command to identify which nodes the containers are actually running on:

   docker ps -a | grep <container-name>

2. If a container is running on a server that has not been added to the machine group, add the IP address of that server to the Machine Groups in the SLS console.

FAQ: Why can't I find the Pod in the container metadata preview, or why can't logs be collected from emptyDir?

Cause: Logtail running in DaemonSet mode cannot directly access the emptyDir temporary storage inside a container. This prevents Logtail from reading log files and extracting container metadata, so the Pod cannot be found in the container metadata Preview.

Solution 1 (Recommended): Redirect application log output to standard output (stdout/stderr).

1. Modify your application to write logs to stdout or stderr instead of files.

2. In the SLS console, configure the collection path to use the standard Kubernetes log path:

   /logtail_host/var/log/pods/<namespace>_<pod-name>-<uid>/<container-name>/*.log

Solution 2: If your application must write logs to files, change the log volume mount from emptyDir to a hostPath or PVC.

1. In your Pod specification, replace the emptyDir volume with a hostPath or PVC definition so that logs are persisted at a path accessible from the host.

2. Adjust the SLS collection path to point to the actual mount path on the host, for example:

   /logtail_host/var/log/your-app/*.log

FAQ: How do I handle file creation failures or permission errors when collecting container logs?

Symptom: An error message indicates that Logtail needs to create a specific empty file inside the container, but file creation fails or a permission error is reported.

Solution:

1. Manually create the empty file specified in the error message inside the container.

2. Set the file permissions to -rw-r--r-- (644):

   chmod 644 <file-path>

3. Restart the container.

Alternative: If the issue persists after the steps above, mount the container log directory to the host machine and configure SLS to collect logs from the corresponding host path. This approach is more stable than in-container file collection.

FAQ: How do I handle the error "parse cri docker line error: invalid CRI log, timestamp not found"?

Cause: Log parsing fails. This error is commonly caused by an incorrect multiline log configuration.

Solution:

1. Check and adjust the start-of-line regular expression, or disable multiline mode:

   • In the YAML configuration: Comment out the multiline configuration section.

   • In the SLS console: Open the Logtail configuration and disable multiline mode.

2. Verify that the K8s Namespace Regex and other format fields are correct. If you need to specify multiple namespaces, ensure they are separated by the correct delimiter.

3. After modifying the YAML configuration, reapply it to make the changes take effect:

   kubectl apply -f <your-config-file>.yaml

Log on to a Logtail container

• Standard Docker

  1. On the host, run the following command to find the Logtail container.

     docker ps | grep logtail

     The output is similar to the following:

     223****6e        registry.cn-hangzhou.aliyuncs.com/log-service/logtail                             "/usr/local/ilogta..."   8 days ago          Up 8 days                               logtail-iba

  2. Run the following command to start a bash shell in the Logtail container.

     docker exec -it 223****6e  bash

     Replace 223****6e with the actual container ID.

• Kubernetes

  1. Run the following command to find the Logtail pod.

     kubectl get po -n kube-system | grep logtail

     The output is similar to the following:

     logtail-ds-****d                                             1/1       Running    0          8d
     logtail-ds-****8                                             1/1       Running    0          8d

  2. Run the following command to log on to the pod.

     kubectl exec -it -n kube-system logtail-ds-****d -- bash

     Replace logtail-ds-****d with the actual pod ID.

View Logtail runtime logs

Logtail stores its logs in the /usr/local/ilogtail/ directory of the Logtail container. The log files are ilogtail.LOG and logtail_plugin.LOG.

1. Log on to the Logtail container. For more information, see Log on to a Logtail container.

2. Go to the /usr/local/ilogtail/ directory.

   cd /usr/local/ilogtail

3. View the ilogtail.LOG and logtail_plugin.LOG files.

   cat ilogtail.LOG
   cat logtail_plugin.LOG

Logtail version, IP address, and startup time

1. On the host, run the following command to view the version, IP address, and startup time of Logtail.

   The information is stored in the /usr/local/ilogtail/app_info.json file of the Logtail container.

   kubectl exec logtail-ds-****k -n kube-system cat /usr/local/ilogtail/app_info.json

   The output is similar to the following:

   {
      "UUID" : "",
      "hostname" : "logtail-****k",
      "instance_id" : "0EB****_172.20.4.2_1517810940",
      "ip" : "172.20.4.2",
      "logtail_version" : "0.16.2",
      "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
      "update_time" : "2018-02-05 06:09:01"
   }

Get troubleshooting information for Pod logs in an ACK cluster

If log collection for a Pod in an ACK cluster is abnormal, you can use the following steps to obtain the basic information needed for self-troubleshooting or for providing to technical support.

Step 1: Find the Logtail or LoongCollector Pod name.

kubectl get pods -n kube-system | grep loongcollector

Step 2: Get the IP address of the Logtail or LoongCollector instance from the app_info.json file.

kubectl exec <pod-name> -n kube-system cat /usr/local/ilogtail/app_info.json

The ip field in the returned JSON output is the Logtail instance IP address.

When seeking support, provide the following information to help with troubleshooting:

• SLS Project name

• Logtail collection configuration name

• Target Pod name

• Target container name