Collect syslog logs

Overview

Linux servers allow you to use syslog agents such as rsyslog to forward local syslog data to a specified server IP address and port. You can configure Logtail for the specified server. In this way, Logtail receives the forwarded syslog data by using the TCP or UDP protocol. It also parses the data based on the specified syslog protocol, and extracts the facility, tag(program), severity, and content fields from the data. The syslog protocol supports the data formats specified by the RFC 3164 and RFC 5424 protocols.

You can configure multiple plug-ins for Logtail. For example, you can use plug-ins to listen on 127.0.0.1:9999 by using both TCP and UDP.

Implementation

After you configure Logtail plug-ins to listen on the specified address and port, you can use Logtail to collect and upload data to Log Service. The data that you can upload includes system logs that are collected by using the rsyslog agent, access logs or error logs that are forwarded by NGINX servers, and logs that are forwarded by syslog clients.

Configure Logtail plug-ins to collect syslog logs

1. Add a forwarding rule for rsyslog.
   1. Modify the /etc/rsyslog.conf file on the server from which syslog logs are collected. Add a forwarding rule at the end of the configuration file.
      Then, rsyslog forwards syslog logs to the specified server IP address and port.

      • You can configure Logtail plug-ins in the server where syslog logs are generated and use this server to upload data to Log Service. To do so, set the IP address to 127.0.0.1 and the port to a non-well-known port.
      • If you want to specify another server to upload syslog logs, set the IP address to the public IP address of the specified server and the port to a non-well-known port.

      The following section describes how to forward syslog logs from a server to Log Service by using TCP and UDP. The IP address is 127.0.0.1 and the port is 9000. For more information about the configurations, visit RSyslog Documentation.

      *. * @@127.0.0.1:9000

   2. Run the following command to restart rsyslog and validate the log forwarding rule:

      sudo service rsyslog restart

2. Log on to the Log Service console.
3. In the Import Data section, select Custom Data Plug-in.
4. In the Specify Logstore step, select the target project and Logstore, and click Next.
   You can also click Create Now to create a project and a Logstore. For more information, see Step 1: Create a project and a Logstore.
5. In the Create Machine Group step, create a machine group.
   • If a machine group is available, click Using Existing Machine Groups.
   • This section uses ECS instances as an example to describe how to create a machine group. To create a machine group, perform the following steps:
     1. Install Logtail on ECS instances. For more information, see Install Logtail on ECS instances.

        If Logtail is installed on the ECS instances, click Complete Installation.

        Note If you need to collect logs from user-created clusters or servers of third-party cloud service providers, you must install Logtail on these servers. For more information, see Install Logtail in Linux or Install Logtail in Windows.
     2. After the installation is complete, click Complete Installation.
     3. On the page that appears, specify the parameters for the machine group. For more information, see Create an IP address-based machine group or Create a custom ID-based machine group.
6. In the Machine Group Settings step, apply the configurations to the machine group.
   Select the created machine group and move the group from Source Server Groups to Applied Server Groups.
7. In the Specify Data Source step, set the Config Name and Plug-in Config parameters.
   • inputs: Required. The Logtail configurations for log collection.
     Note You can configure only one type of data source in the inputs field.
   • processors: Optional. The Logtail configurations for data processing. You can configure one or more processing methods in the processors field. For more information, see Process data.

   The following sample code shows how to use UDP and TCP to listen on 127.0.0.1:9000:

   {
        "inputs": [
            {
                "type": "service_syslog",
                "detail": {
                    "Address": "tcp://127.0.0.1:9000",
                    "ParseProtocol": "rfc3164"
                }
            },
            {
                "type": "service_syslog",
                "detail": {
                    "Address": "udp://127.0.0.1:9001",
                    "ParseProtocol": "rfc3164"
                }
            }
        ]
    }

   Parameter	Type	Required	Description
   type	String	Yes	The type of the data source. Set the value to service_syslog.
   Address	String	No	The protocol that the Logtail plug-in uses, and the IP address and port that the Logtail plug-in listens on to obtain log data. The value of the parameter is in the [tcp/udp]://[ip]:[port] format. Default value: tcp://127.0.0.1:9999. Note The specified protocol, address, and port must be the same as those specified in the forwarding rule of the rsyslog configuration file. If the server on which Logtail is installed has multiple IP addresses, you can set the IP address to 0.0.0.0. This address indicates that the plug-in listens on all IP addresses of the server.
   ParseProtocol	String	No	The protocol that is used to parse logs. Default value: null. This value indicates that logs are not parsed. Other valid values include: rfc3164: The RFC 3164 protocol is used to parse logs. rfc5424: The RFC 5424 protocol is used to parse logs. auto: The Logtail plug-in selects a protocol to parse logs based on the log content.
   IgnoreParseFailure	Boolean	No	Specifies the operation if a log fails to be parsed. Default value: true. This value indicates that logs are not parsed and are included in the content field. The value false indicates that logs are dropped if they fail to be parsed.

8. In the Configure Query and Analysis step, configure the indexes.
   Indexes are configured by default. You can re-configure the indexes based on your business requirements. For more information, see Enable and configure the index feature for a Logstore.

   Note

   • You must configure Full Text Index or Field Search. If you configure both of them, the settings of Field Search are applied.
   • If the data type of index is long or double, the Case Sensitive and Delimiter settings are unavailable.

Configure Logtail plug-ins to collect NGINX logs

NGINX servers allow you to forward access logs to specified IP addresses and ports by using the syslog protocol. You can create Logtail configurations to convert NGINX logs such as access logs to syslog logs. After you convert the access logs to syslog logs, you can send the syslog logs to Log Service.

1. Add a forwarding rule for the NGINX server.
   1. Find the nginx.conf file on the NGINX server, and add a forward rule to the end of the file. For more information, visit NGINX Beginner's Guide.
      The following script provides a sample forwarding rule:

      http {
          ...
      
          # Add this line.
          access_log syslog:server=127.0.0.1:9000,facility=local7,tag=nginx,severity=info combined;
      
          ...
      }
      								

   2. Run the following command to restart the NGINX service and validate the configuration:

      sudo service nginx restart

2. Create a Logtail configuration file. For more information, see Configure Logtail plug-ins to collect syslog logs.