This topic describes how to configure Logtail in the Log Service console to collect
syslog logs.
Prerequisites
Logtail is installed on the server that you use to collect syslog logs. For more information,
see
Install Logtail in Linux or
Install Logtail in Windows.
Note Servers that run Linux support Logtail 0.16.13 or later. Servers that run Windows support Logtail 1.0.0.8 or later.
Overview
Linux servers allow you to use syslog agents such as rsyslog to forward local syslog
data to a specified server IP address and port. You can configure Logtail for the
specified server. In this way, Logtail receives the forwarded syslog data by using
the TCP or UDP protocol. It also parses the data based on the specified syslog protocol,
and extracts the facility, tag(program), severity, and content fields from the data.
The syslog protocol supports the data formats specified by the RFC 3164 and RFC 5424 protocols.
You can configure multiple plug-ins for Logtail. For example, you can use plug-ins
to listen on 127.0.0.1:9999 by using both TCP and UDP.
Implementation
After you configure Logtail plug-ins to listen on the specified address and port,
you can use Logtail to collect and upload data to Log Service. The data that you can
upload includes system logs that are collected by using the rsyslog agent, access
logs or error logs that are forwarded by
NGINX servers, and logs that are forwarded by syslog clients.

Configure Logtail plug-ins to collect syslog logs
- Add a forwarding rule for rsyslog.
- Modify the /etc/rsyslog.conf file on the server from which syslog logs are collected. Add a forwarding rule at
the end of the configuration file.
Then, rsyslog forwards syslog logs to the specified server IP address and port.
- You can configure Logtail plug-ins in the server where syslog logs are generated and
use this server to upload data to Log Service. To do so, set the IP address to 127.0.0.1
and the port to a non-well-known port.
- If you want to specify another server to upload syslog logs, set the IP address to
the public IP address of the specified server and the port to a non-well-known port.
The following section describes how to forward syslog logs from a server to Log Service
by using TCP and UDP. The IP address is 127.0.0.1 and the port is 9000. For more information
about the configurations, visit
RSyslog Documentation.
*. * @@127.0.0.1:9000
- Run the following command to restart rsyslog and validate the log forwarding rule:
sudo service rsyslog restart
- Log on to the Log Service console.
- In the Import Data section, select Custom Data Plug-in.
- In the Specify Logstore step, select the target project and Logstore, and click Next.
- In the Create Machine Group step, create a machine group.
- If a machine group is available, click Using Existing Machine Groups.
- This section uses ECS instances as an example to describe how to create a machine
group. To create a machine group, perform the following steps:
- Install Logtail on ECS instances. For more information, see Install Logtail on ECS instances.
If Logtail is installed on the ECS instances, click Complete Installation.
- After the installation is complete, click Complete Installation.
- On the page that appears, specify the parameters for the machine group. For more information,
see Create an IP address-based machine group or Create a custom ID-based machine group.
- In the Machine Group Settings step, apply the configurations to the machine group.
Select the created machine group and move the group from Source Server Groups to Applied Server Groups.
- In the Specify Data Source step, set the Config Name and Plug-in Config parameters.
- inputs: Required. The Logtail configurations for log collection.
Note You can configure only one type of data source in the inputs field.
- processors: Optional. The Logtail configurations for data processing. You can configure one
or more processing methods in the processors field. For more information, see Process data.
The following sample code shows how to use UDP and TCP to listen on 127.0.0.1:9000:
{
"inputs": [
{
"type": "service_syslog",
"detail": {
"Address": "tcp://127.0.0.1:9000",
"ParseProtocol": "rfc3164"
}
},
{
"type": "service_syslog",
"detail": {
"Address": "udp://127.0.0.1:9001",
"ParseProtocol": "rfc3164"
}
}
]
}
Parameter |
Type |
Required |
Description |
type |
String |
Yes |
The type of the data source. Set the value to service_syslog.
|
Address |
String |
No |
The protocol that the Logtail plug-in uses, and the IP address and port that the Logtail
plug-in listens on to obtain log data. The value of the parameter is in the [tcp/udp]://[ip]:[port] format. Default value: tcp://127.0.0.1:9999.
Note
- The specified protocol, address, and port must be the same as those specified in the
forwarding rule of the rsyslog configuration file.
- If the server on which Logtail is installed has multiple IP addresses, you can set
the IP address to 0.0.0.0. This address indicates that the plug-in listens on all IP addresses of the server.
|
ParseProtocol |
String |
No |
The protocol that is used to parse logs. Default value: null. This value indicates
that logs are not parsed. Other valid values include:
- rfc3164: The RFC 3164 protocol is used to parse logs.
- rfc5424: The RFC 5424 protocol is used to parse logs.
- auto: The Logtail plug-in selects a protocol to parse logs based on the log content.
|
IgnoreParseFailure |
Boolean |
No |
Specifies the operation if a log fails to be parsed. Default value: true. This value indicates that logs are not parsed and are included in the content field.
The value false indicates that logs are dropped if they fail to be parsed.
|
- In the Configure Query and Analysis step, configure the indexes.
Indexes are configured by default. You can re-configure the indexes based on your
business requirements. For more information, see
Enable and configure the index feature for a Logstore.
Note
- You must configure Full Text Index or Field Search. If you configure both of them,
the settings of Field Search are applied.
- If the data type of index is long or double, the Case Sensitive and Delimiter settings
are unavailable.
Configure Logtail plug-ins to collect NGINX logs
NGINX servers allow you to forward access logs to specified IP addresses and ports
by using the syslog protocol. You can create Logtail configurations to convert NGINX
logs such as access logs to syslog logs. After you convert the access logs to syslog
logs, you can send the syslog logs to Log Service.
- Add a forwarding rule for the NGINX server.
- Find the nginx.conf file on the NGINX server, and add a forward rule to the end of the file. For more
information, visit NGINX Beginner's Guide.
The following script provides a sample forwarding rule:
http {
...
# Add this line.
access_log syslog:server=127.0.0.1:9000,facility=local7,tag=nginx,severity=info combined;
...
}
- Run the following command to restart the NGINX service and validate the configuration:
sudo service nginx restart
- Create a Logtail configuration file. For more information, see Configure Logtail plug-ins to collect syslog logs.
What to do next
After Logtail collects syslog logs to Log Service, you can view the logs in the Log
Service console.
Field |
Description |
_hostname_ |
The hostname. If a hostname is not provided in the log entry, the hostname of the
current host is obtained.
|
_program_ |
The tag field in the syslog protocol. |
_priority_ |
The priority field in the syslog protocol. |
_facility_ |
The facility field in the syslog protocol. |
_severity_ |
The severity field in the syslog protocol. |
_unixtimestamp_ |
The timestamp of the log entry. |
_content_ |
The log content. If the log entry fails to be parsed, the unparsed log entry is included
in the content field and uploaded to Log Service.
|
_ip_ |
The IP address of the current host. |