Collect Linux Syslog with Logtail | SLS - Simple Log Service

How it works

On a Linux server, a Syslog agent such as rsyslog forwards local Syslog data to a specified IP address and port. Logtail listens on that address and port, receives data over TCP or UDP, and parses it per the configured Syslog protocol to extract fields such as facility, tag (program), severity, and content.

Syslog supports the protocols defined in RFC 3164 and RFC 5424.

Prerequisites

A project and a Standard logstore are created. For more information, see Manage projects and Create a basic logstore.
This feature supports only Linux and requires the latest Logtail version. Install Logtail on a Linux server.

Procedure

The following steps use an ECS instance as an example.

Log on to the Simple Log Service console.
In the Import Data section, select Custom Data Plug-in.
Select the project and logstore. Then, click Next.
Create a machine group.
- If a machine group is available, click Use Existing Machine Groups.
- If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
  1. On the ECS Instance tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.
    
    For more information, see Install Logtail on ECS instances.
    
    Important
    If your server is an ECS instance that belongs to a different Alibaba Cloud account from Simple Log Service, a server in a data center, or a server from a third-party cloud service provider, you must manually install Logtail on the server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.
  2. After you confirm the settings, click OK.
  3. After Logtail is installed, click Complete Installation.
  4. In the Create Machine Group step, configure the Name parameter and click Next.
    
    Simple Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.
Confirm that the machine group is displayed in the Applied Server Groups section and click Next.

Important
If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

On the Configure Data Source tab, set Configuration Name, copy and paste the following script into the Plug-in Configuration field, and then click Next.

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "tcp://0.0.0.0:9000",
                 "ParseProtocol": "rfc3164"
             }
         },
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://0.0.0.0:9001",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Parameters:

Parameter	Type	Required	Description
`type`	string	Yes	The data source type. Set to `service_syslog`.
`Address`	string	No	The listening endpoint. Format: `[tcp/udp]://[ip]:[port]`. Default: `tcp://127.0.0.1:9999` (local server only). Note The protocol, address, and port number specified in the Logtail configuration must match the forwarding rule in your rsyslog configuration file. If the server where Logtail is installed has multiple IP addresses that can receive logs, you can set the address to `0.0.0.0` to listen on all IP addresses of the server.
`ParseProtocol`	string	No	The parsing protocol. Default: empty (no parsing). Valid values: Empty: Logs are not parsed. rfc3164: Logs are parsed using the RFC 3164 protocol. rfc5424: Logs are parsed using the RFC 5424 protocol. auto: A suitable protocol is automatically selected based on the log content.
`IgnoreParseFailure`	boolean	No	Action when parsing fails. Default: `true` (skip parsing and write the raw log to the `content` field). Set to `false` to drop unparseable logs.

On the Query and Analysis Configurations tab, click Automatic Index Generation. No data is collected at this point. Click Next to complete the Logtail configuration.
Log on to the ECS instance that generates the Syslog, and then configure rsyslog forwarding.
1. On the server where Syslog is generated, modify the rsyslog configuration file at /etc/rsyslog.conf and add a forwarding rule to the end of the file.
  After adding the rule, rsyslog forwards Syslog to the specified address and port.
  - If you collect Syslog from the same server that runs Logtail, set the forwarding address to 127.0.0.1 and use the same port specified in your Logtail configuration.
  - If you collect Syslog from a remote server, set the forwarding address to the public IP address of the server running Logtail and use the same port specified in your Logtail configuration.
  For example, the following rule forwards all logs to 127.0.0.1:9000 over TCP. The full syntax is covered in the RSyslog documentation.
```
*.* @@127.0.0.1:9000
```
2. Run the following command to restart rsyslog for the forwarding rule to take effect.
```
sudo service rsyslog restart
```
View the collected Syslog data in the destination Logstore. Quick start for query and analysis.

Log fields

Field	Description
`_hostname_`	Defaults to the current host's hostname if not provided in the log.
`_program_`	The tag field from the Syslog protocol.
`_priority_`	The priority field from the Syslog protocol.
`_facility_`	The facility field from the Syslog protocol.
`_severity_`	The severity field from the Syslog protocol.
`_unixtimestamp_`	The timestamp of the log.
`_content_`	The content of the log. If parsing fails, this field contains the full content of the raw log.
`_ip_`	The IP address of the current host.
`_client_ip_`	The IP address of the client that sends the log.

More examples

Nginx logs

Nginx natively supports forwarding access logs via the Syslog protocol. To centrally collect all server data, including Nginx access logs, in SLS via Syslog, create a Logtail configuration.

Create a Logtail configuration as described in Collect Linux Syslog.

The following code shows a sample Plug-in Configuration script.

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://127.0.0.1:9001",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Add a forwarding rule for Nginx.

Add a forwarding rule to nginx.conf. The official Nginx documentation covers the configuration structure.

Example:

server {
    ...

    # Add this line.
    access_log syslog:server=127.0.0.1:9001,facility=local7,tag=nginx,severity=info combined;

    ...
}

Run the following command to restart the Nginx service for the configuration to take effect.
```
sudo service nginx restart
```

Firewall logs

Firewalls typically support forwarding logs via the Syslog protocol. To centrally collect firewall logs in SLS (for example, to build a SIEM), create a Logtail configuration.

Create a Logtail configuration as described in Collect Linux Syslog.

The following code shows a sample Plug-in Configuration script.

{
     "inputs": [
         {
             "type": "service_syslog",
             "detail": {
                 "Address": "udp://0.0.0.0:9001",
                 "ParseProtocol": "rfc3164"
             }
         }
     ]
 }

Add a forwarding rule per your firewall's documentation. In this example, the Logtail server IP is 10.20.30.40.

Troubleshooting

If no data appears after configuring Logtail, run the logger command to send test data to the local Logtail instance. This isolates whether the issue is with the data source, the network, or Logtail.

If the local Logtail instance receives the data, the issue is likely with the data source or the network.

logger -n localhost -P 9000 -T "This is a TCP syslog message"
logger -n localhost -P 9001 -d "This is a UDP syslog message"

Troubleshoot Logtail log collection failures.