Troubleshoot log collection failures - Simple Log Service

If Simple Log Service (SLS) is not collecting log data, work through the following checks in order. Each section targets a specific component in the collection pipeline.

Quick diagnostic checklist

Eliminate the most common causes first:

[ ] Is the rsyslog service running and forwarding logs to the correct port?
[ ] Is the Logtail agent installed and running on the Elastic Compute Service (ECS) instance?
[ ] Does the machine group show a heartbeat in the SLS console?
[ ] Are there any collection errors reported in the SLS console?
[ ] Can you see log data in Preview? If so, the issue may be with indexes rather than collection.

If any check fails, go to the corresponding section below.

Check the rsyslog service

If logs are forwarded through rsyslog before Logtail collects them, verify the rsyslog configuration and service status.

Verify the rsyslog configuration

Log on to the ECS instance and check the syslog.conf configuration file. A typical configuration:

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1     # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g       # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on     # save messages to disk on shutdown
$ActionQueueType LinkedList       # run asynchronously
$ActionResumeRetryCount -1        # infinite retries if host is down

$template ALI_LOG_FMT,"0.1 sys_tag %timegenerated:::date-unixtimestamp% %fromhost-ip%
%hostname% %pri-text% %protocol-version% %app-name% %procid% %msgid%
%msg:::drop-last-lf%\n"

*.*  @@<ip>:<port>;ALI_LOG_FMT

The template content is enclosed in double quotation marks (""). The example above is split across multiple lines for readability.

Element	Description
`$template`	Defines a message template. `ALI_LOG_FMT` is the template name.
`0.1`	Log format version. Logtail uses this version to parse user-defined fields.
`sys_tag`	Data tag used to identify the project or Logstore in the SLS console.
`%timegenerated:::date-unixtimestamp%`	Timestamp of the log entry.
`%fromhost-ip%`	IP address of the server from which the log is collected. If this value is `127.0.0.1`, SLS replaces it with the peer IP address of the TCP socket.
`%hostname%` through `%msgid%`	User-defined fields. Fields in brackets are optional.
`%msg:::drop-last-lf%`	Body of the log message.
`@@`	Forward over TCP. Use a single `@` for UDP.
`<ip>:<port>`	Destination IP address and port for log forwarding.
`.`	The first `` matches all message types. The second `` matches all message levels.

Verify the rsyslog service status

Run the following command:

service rsyslog status

Verify the forwarding ports

Run the following command to check whether rsyslog is forwarding data to the expected port (port 11111 in this example):

netstat -antpul | grep 11111

Expected output: Two sockets monitoring and forwarding data to port 11111. For example, one socket monitors port 53552 and forwards to port 11111, while another monitors port 53554 and forwards to port 11111. Each forwarding rule in syslog.conf creates a corresponding socket.

In the output, also verify:

Three Logtail processes are running, with two in the ESTABLISHED state for monitoring.
Two rsyslog processes are in the ESTABLISHED state for monitoring.

Check the Logtail agent

Run the following command on the ECS instance to check the Logtail status:
```
/etc/init.d/ilogtaild status
```
Check the ilogtail.LOG file for errors reported by the Logtail agent.

Check the machine group status

Check the heartbeat status of your machine group. For details, see What do I do if a Logtail machine group has no heartbeats?

A machine group with no heartbeat means Logtail on that machine cannot communicate with SLS.

Check errors in the SLS console

Log on to the Simple Log Service console.
View reported collection errors and identify the issue. For details, see How do I view Logtail collection errors?
Click Preview to check whether log data is being collected.
- If log data appears in Preview but you cannot query it, indexes are not enabled or the index configuration is invalid. For details, see Query and analyze logs.
- If no log data appears, continue to the next section.

Still not working?

If all checks above pass but log collection still fails, gather the following information and submit a support ticket:

Logtail version and the output of /etc/init.d/ilogtaild status
Relevant entries from ilogtail.LOG
The rsyslog configuration file (syslog.conf)
The output of the netstat command above