This topic describes how to create a Logtail configuration in the Log Service console to collect syslogs.

Prerequisites

Logtail V0.16.13 or later is installed on a Linux server, or Logtail V1.0.0.8 or later is installed on a Windows server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.

Overview

Linux servers allow you to use syslog agents such as rsyslog to forward local syslogs to the IP address and port of a specified server. After you apply a Logtail configuration to the specified server, the Logtail plug-in specified in the configuration receives the forwarded syslogs over TCP or UDP. The plug-in also parses the syslogs based on the specified syslog protocol, and extracts the facility, tag(program), severity, and content fields from the syslogs. The syslog protocols defined in RFC 3164 and RFC 5424 are supported.

You can configure multiple Logtail plug-ins based on your business requirements. For example, you can configure two Logtail plug-ins to listen on 127.0.0.1:9000 over both TCP and UDP.

Implementation

After you configure Logtail plug-ins to listen on a specified address and port, Logtail collects and sends data to Log Service. The data includes the system logs that are collected by using rsyslog, the access logs or error logs that are forwarded by NGINX, and the logs that are forwarded by syslog clients. Implementation

Configure Logtail plug-ins to collect syslogs

  1. Add a forwarding rule for rsyslog.
    1. Modify the /etc/rsyslog.conf configuration file of rsyslog on the server from which you want to collect syslogs. Add a forwarding rule to the end of the configuration file.
      After the forwarding rule is added, rsyslog forwards syslogs to a specified IP address and port.
      • If Logtail resides on the syslog server, you must specify the IP address 127.0.0.1 and a non-well-known port that is unoccupied in the forwarding rule.
      • If Logtail resides on a different server from the syslog server, you must specify the public IP address of the different server and a non-well-known port that is unoccupied in the forwarding rule.
      The following example shows a forwarding rule, which allows all syslogs to be forwarded to 127.0.0.1:9000 over TCP. For more information about the configuration file, see RSyslog Documentation.
      *.* @@127.0.0.1:9000
    2. Run the following command to restart rsyslog and validate the forwarding rule:
      sudo service rsyslog restart
  2. Log on to the Log Service console.
  3. In the Import Data section, select Custom Data Plug-in.
  4. Select the project and Logstore. Then, click Next.
  5. Create a machine group.
    • If a machine group is available, click Use Existing Machine Groups.
    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
      1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important If you want to collect logs from an ECS instance that belongs to a different Alibaba Cloud account, a server in an on-premises data center, or a server of a third-party cloud service provider, you must manually install Logtail. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server.

        After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.

      2. After Logtail is installed, click Complete Installation.
      3. In the Create Machine Group step, configure the Name parameter and click Next.

        Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

  6. Select the new machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
    Important If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  7. In the Specify Data Source step, configure the Config Name and Plug-in Config parameters. Then, click Next.
    • inputs specifies the collection configurations of your data source. This parameter is required.
      Important You can specify only one type of data source in the inputs parameter.
    • processors specifies the processing configurations that are used to parse data. You can extract fields, extract log time, desensitize data, and filter logs. This parameter is optional. You can specify one or more processing methods. For more information, see Overview.
    The following example shows how to configure Logtail plug-ins to listen on 127.0.0.1 over UDP and TCP:
    {
         "inputs": [
             {
                 "type": "service_syslog",
                 "detail": {
                     "Address": "tcp://127.0.0.1:9000",
                     "ParseProtocol": "rfc3164"
                 }
             },
             {
                 "type": "service_syslog",
                 "detail": {
                     "Address": "udp://127.0.0.1:9001",
                     "ParseProtocol": "rfc3164"
                 }
             }
         ]
     }
    ParameterTypeRequiredDescription
    typestringYesThe type of the data source. Set the value to service_syslog.
    AddressstringNoThe listening protocol, address, and port that are used by a Logtail plug-in. The plug-in listens on and obtains data based on the Logtail configuration. Specify the value in the [tcp/udp]://[ip]:[port] format. Default value: tcp://127.0.0.1:9999.
    Note
    • The listening protocol, address, and port that you specify must be the same as those specified in the forwarding rule that is added to the configuration file of rsyslog.
    • If the Logtail server uses multiple IP addresses to receive data, set the IP address to 0.0.0.0. This address indicates that the plug-in listens on all the IP addresses of the server.
    ParseProtocolstringNoThe protocol that is used to parse syslogs. By default, this parameter is empty, which indicates that syslogs are not parsed. Valid values:
    • rfc3164: The RFC 3164 protocol is used to parse syslogs.
    • rfc5424: The RFC 5424 protocol is used to parse syslogs.
    • auto: The plug-in automatically selects a protocol based on the content of syslogs.
    IgnoreParseFailurebooleanNoSpecifies whether to perform an operation on a syslog after the syslog fails to be parsed. Default value: true. This value indicates that the system does not parse the syslog and adds the syslog to the content field. If you set the value to false, the syslog is discarded after it fails to be parsed.
  8. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
    Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
  9. Click Log Query. You are redirected to the query and analysis page of your Logstore.
    You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Configure Logtail plug-ins to collect NGINX logs

NGINX servers allow you to forward access logs to specified IP addresses and ports by using the syslog protocol. If you want to deliver all data of a server as syslogs to Log Service, you can create a Logtail configuration to collect the data. The data includes NGINX access logs.

  1. Add a forwarding rule for NGINX.
    1. Find the nginx.conf file on the NGINX server and add a forwarding rule to the end of the file. For more information, see NGINX Beginner's Guide.
      The following sample code provides an example of a forwarding rule:
      http {
          ...
      
          # Add this line.
          access_log syslog:server=127.0.0.1:9000,facility=local7,tag=nginx,severity=info combined;
      
          ...
      }
                                      
    2. Run the following command to restart the NGINX service and validate the forwarding rule:
      sudo service nginx restart
  2. Create a Logtail configuration. For more information, see Configure Logtail plug-ins to collect syslogs.

Troubleshooting

If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can troubleshoot the errors based on the instructions that are provided in What do I do if errors occur when I use Logtail to collect logs?

Sample log

The following figure shows a sample syslog.

Validate the Logtail configuration
FieldDescription
_hostname_The hostname. If no hostname is included in the log, the hostname of the current host is obtained.
_program_The tag field in the syslog protocol.
_priority_The priority field in the syslog protocol.
_facility_The facility field in the syslog protocol.
_severity_The severity field in the syslog protocol.
_unixtimestamp_The timestamp of the log.
_content_The content of the log. If the log fails to be parsed, this field contains the content of the raw log.
_ip_The IP address of the current host.
_client_ip_The IP address of the syslog client that transfers logs.