Simple Log Service:Collect Linux syslogs

Implementation

Linux servers allow you to use syslog agents such as rsyslog to forward on-premises syslogs to a server by using a specific IP address and port. Logtail listens to the IP address and port, receives the forwarded syslogs over TCP or UDP, parses the syslogs based on a specific syslog protocol, and then extracts fields such as facility, tag (program), severity, and content from the syslogs.

The syslog protocols defined in RFC 3164 and RFC 5424 are supported. For more information, see RFC3164 and RFC5424.

Procedure

In this topic, syslogs are collected from an Elastic Compute Service (ECS) instance.

1. Log on to the Simple Log Service console.

2. In the Import Data dialog box, select Custom Data Plug-in.

3. Select the project and Logstore. Then, click Next.

4. Create a machine group.

   • If a machine group is available, click Use Existing Machine Groups.

   • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.

     1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important

        If your server is an ECS instance that belongs to a different Alibaba Cloud account from Simple Log Service, a server in a data center, or a server from a third-party cloud service provider, you must manually install Logtail on the server. For more information, see Install Logtail on a Linux server or Install Logtail on a Windows server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.

     2. After you confirm the settings, click OK.

     3. After Logtail is installed, click Complete Installation.

     4. In the Create Machine Group step, configure the Name parameter and click Next.

        Simple Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

5. Confirm that the machine group is displayed in the Applied Server Groups section and click Next.

   Important

   If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

6. In the Configure Data Source step of the Import Data wizard, configure the Configuration Name parameter, copy and paste the following configuration script to the Plug-in Configuration field, and then click Next.

   {
        "inputs": [
            {
                "type": "service_syslog",
                "detail": {
                    "Address": "tcp://0.0.0.0:9000",
                    "ParseProtocol": "rfc3164"
                }
            },
            {
                "type": "service_syslog",
                "detail": {
                    "Address": "udp://0.0.0.0:9001",
                    "ParseProtocol": "rfc3164"
                }
            }
        ]
    }

   The following table describes the parameters in the configuration script.

   Parameter	Type	Required	Description
   type	string	Yes	The type of the data source. The value is fixed as service_syslog.
   Address	string	No	The listener protocol, IP address, and port of Logtail. Logtail listens to log sources and obtains logs based on your Logtail configuration. The value is in the [tcp/udp]://[ip]:[port] format. Default value: tcp://127.0.0.1:9999, which indicates that Logtail receives only logs that are forwarded to the Logtail server. Note The listener protocol, IP address, and port that you specify in your Logtail configuration must be the same as those specified in the forwarding rule that is added to the configuration file of rsyslog. If the server on which Logtail is installed uses multiple IP addresses to receive logs, set Address to 0.0.0.0, which indicates that Logtail listens to all IP addresses of the server.
   ParseProtocol	string	No	The protocol that you want Logtail to use to parse logs. This parameter is left empty by default, which indicates that logs are not parsed. Valid values: Empty: Logtail does not parse syslogs. rfc3164: The RFC 3164 protocol is used to parse logs. rfc5424: The RFC 5424 protocol is used to parse logs. auto: Logtail automatically selects a protocol based on the content of logs.
   IgnoreParseFailure	boolean	No	Specifies whether to perform an operation on a log after the log fails to be parsed. Valid values: true and false. Default value: true, which indicates that the system does not parse the log and adds the log to the content field. If you set this parameter to false, the system discards the log after the log fails to be parsed.

7. In the Query and Analysis Configurations step of the Import Data wizard, click Automatic Index Generation. After you configure this setting, no data is collected. To collect data, click Next to complete the Logtail configuration.

8. Log on to the ECS instance from which you want to collect syslogs and configure rsyslog forwarding.

   1. Modify the /etc/rsyslog.conf configuration file of rsyslog on the ECS instance. Add a forwarding rule to the end of the configuration file.

      After the forwarding rule is added, rsyslog forwards syslogs to a specific IP address and port.

      • If Logtail resides on the ECS instance, you must specify the IP address 127.0.0.1 and the port that you specify in your Logtail configuration in the forwarding rule.

      • If Logtail resides on a different server from the ECS instance, you must specify the public IP address of the server and the port that you specify in your Logtail configuration in the forwarding rule.

      The following sample code provides an example of a forwarding rule, which allows all syslogs to be forwarded to 127.0.0.1:9000 over TCP. For more information about the configuration file, see RSyslog Documentation.

      *.* @@127.0.0.1:9000

   2. Run the following command to restart rsyslog and validate the forwarding rule:

      sudo service rsyslog restart

9. Log on to the Simple Log Service console and view the collected syslogs in the required Logstore. For more information, see Guide to log query and analysis.

Log field descriptions

Examples

Troubleshooting

If no data is displayed on the preview page or the query page after logs are collected by using Logtail, you can run logger commands to send data to local Logtail. Then, you can check whether the issue occurred due to the writer, network, or Logtail.

If local Logtail can receive data, the issue occurred due to the writer or the network.

logger -n localhost -P 9000 -T "This is a TCP syslog message"
logger -n localhost -P 9001 -d "This is a UDP syslog message"

For more information, see What do I do if errors occur when I use Logtail to collect logs?