This topic describes how to create a Logtail configuration in the Log Service console to collect syslogs.

Prerequisites

Logtail is installed on a server. For ease of understanding, this server is referred to as the Logtail server in this topic. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.
Note Linux servers support Logtail V0.16.13 or later. Windows servers support Logtail V1.0.0.8 or later.

Overview

Linux servers allow you to use syslog agents such as rsyslog to forward local syslogs to the IP address and port of a specified server. After you apply a Logtail configuration to the specified server, the Logtail plug-in specified in the configuration receives the forwarded syslogs over TCP or UDP. The plug-in also parses the syslogs based on the specified syslog protocol, and extracts the facility, tag(program), severity, and content fields from the syslogs. The syslog protocols defined in RFC 3164 and RFC 5424 are supported.

You can configure multiple Logtail plug-ins based on your business requirements. For example, you can configure two Logtail plug-ins to listen on 127.0.0.1:9000 over both TCP and UDP.

Principle

After you configure Logtail plug-ins to listen on a specified address and port, Logtail collects and sends data to Log Service. The data includes the system logs that are collected by using rsyslog, the access logs or error logs that are forwarded by NGINX, and the logs that are forwarded by syslog clients. Principle

Configure Logtail plug-ins to collect syslogs

  1. Add a forwarding rule for rsyslog.
    1. Modify the /etc/rsyslog.conf configuration file of rsyslog on the server from which you want to collect syslogs. (For ease of understanding, this server is referred to as the syslog server in this section.) Add a forwarding rule to the end of the configuration file.
      After the forwarding rule is added, rsyslog forwards syslogs to a specified IP address and port.
      • If Logtail resides on the syslog server, you must specify the IP address 127.0.0.1 and a non-well-known port that is unoccupied in the forwarding rule.
      • If Logtail resides on a different server from the syslog server, you must specify the public IP address of the different server and a non-well-known port that is unoccupied in the forwarding rule.
      The following example shows a forwarding rule, which allows all syslogs to be forwarded to 127.0.0.1:9000 over TCP. For more information about the configuration file, see RSyslog Documentation.
      *.* @@127.0.0.1:9000
    2. Run the following command to restart rsyslog and validate the forwarding rule:
      sudo service rsyslog restart
  2. Log on to the Log Service console.
  3. In the Import Data section, click Custom Data Plug-in.
  4. Select the project and Logstore. Then, click Next.
  5. Create a machine group.
    • If a machine group is available, click Use Existing Machine Groups.
    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
      1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Execute Now.

        For more information, see Install Logtail on ECS instances.

        Note If you want to collect logs from an ECS instance that belongs to a different Alibaba Cloud account, a server in an on-premises data center, or a server of a third-party cloud service provider, you must manually install Logtail. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier on the server. For more information, see Configure a user identifier.
      2. After Logtail is installed, click Complete Installation.
      3. In the Create Machine Group step, configure Name and click Next.

        Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom ID-based machine group.

  6. Select the new machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
    Notice If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  7. In the Specify Data Source step, configure Config Name and Plug-in Config. Then, click Next.
    • inputs is required and is used to configure the data collection settings for the Logtail configuration. You must configure inputs based on your data source.
      Note You can specify only one type of data source in inputs.
    • processors is optional and is used to configure the data processing settings for the Logtail configuration. You can specify one or more data processing methods in processors. For more information, see Overview.
    The following example shows how to configure Logtail plug-ins to listen on 127.0.0.1 over both UDP and TCP:
    {
         "inputs": [
             {
                 "type": "service_syslog",
                 "detail": {
                     "Address": "tcp://127.0.0.1:9000",
                     "ParseProtocol": "rfc3164"
                 }
             },
             {
                 "type": "service_syslog",
                 "detail": {
                     "Address": "udp://127.0.0.1:9001",
                     "ParseProtocol": "rfc3164"
                 }
             }
         ]
     }
    Parameter Type Required Description
    type string Yes The type of the data source. Set the value to service_syslog.
    Address string No The listening protocol, address, and port that are used by a Logtail plug-in. The plug-in listens on and obtains data based on the Logtail configuration. Specify the value in the [tcp/udp]://[ip]:[port] format. If you do not configure this parameter, the default value tcp://127.0.0.1:9999 is used.
    Note
    • The listening protocol, address, and port that you specify must be the same as those specified in the forwarding rule that is added to the configuration file of rsyslog.
    • If the Logtail server uses multiple IP addresses to receive data, set the IP address to 0.0.0.0. This address indicates that the plug-in listens on all the IP addresses of the server.
    ParseProtocol string No The protocol that is used to parse syslogs. By default, this parameter is empty, which indicates that syslogs are not parsed. Valid values:
    • rfc3164: The RFC 3164 protocol is used to parse syslogs.
    • rfc5424: The RFC 5424 protocol is used to parse syslogs.
    • auto: The plug-in automatically selects a protocol based on the content of syslogs.
    IgnoreParseFailure boolean No The operation that is performed after a syslog fails to be parsed. If you do not configure this parameter, the default value true is used. This value indicates that a syslog that fails to be parsed is directly included in the content field that is returned. If you set the value to false, a syslog is discarded after it fails to be parsed.

Configure Logtail plug-ins to collect NGINX logs

NGINX servers allow you to directly forward access logs to specified IP addresses and ports by using the syslog protocol. If you want to deliver all the data of a server as syslogs to Log Service, you can create a Logtail configuration to collect the data. The data includes NGINX access logs.

  1. Add a forwarding rule for NGINX.
    1. Find the nginx.conf file on the NGINX server and add a forward rule to the end of the file. For more information, see NGINX Beginner's Guide.
      The following sample code provides an example of a forwarding rule:
      http {
          ...
      
          # Add this line.
          access_log syslog:server=127.0.0.1:9000,facility=local7,tag=nginx,severity=info combined;
      
          ...
      }
                                      
    2. Run the following command to restart the NGINX service and validate the forwarding rule:
      sudo service nginx restart
  2. Create a Logtail configuration. For more information, see Configure Logtail plug-ins to collect syslogs.

What to do next

After Logtail collects and sends syslogs to Log Service, you can view the logs in the Log Service console.

syslog
Field Description
_hostname_ The hostname. If no hostname is included in the log, the hostname of the current host is obtained.
_program_ The tag field in the syslog protocol.
_priority_ The priority field in the syslog protocol.
_facility_ The facility field in the syslog protocol.
_severity_ The severity field in the syslog protocol.
_unixtimestamp_ The timestamp of the log.
_content_ The content of the log. If the log fails to be parsed, this field contains the content of the log that is not parsed.
_ip_ The IP address of the current host.
_client_ip_ The IP address of Logtail that transfers the log.