You can use Logtail to collect Linux systemd journal logs from binary files. This topic describes how to create a Logtail configuration to collect systemd journal logs in the Log Service console.

Prerequisites

Logtail V0.16.18 or later is installed on a Linux server. For more information, see Install Logtail on a Linux server.

Introduction

systemd is a system and service manager for Linux operating systems. systemd runs as an init process with PID 1. systemd replaces Upstart as the init system to boot and maintain services in user space. systemd manages the logs of all units in a centralized manner. The configuration file is /etc/systemd/journald.conf, and the logs include kernel and application logs.

Note The operating system on which systemd runs must support the journal log format.

Features

  • Allows you to specify the position for initial collection. Checkpoints are automatically stored for subsequent data collection. The process of systemd is not affected when applications are restarted.
  • Allows you to filter units.
  • Allows you to collect kernel logs.
  • Supports automatic parsing of log levels.
  • Allows you to run systemd as a container to collect journal logs from hosts. This feature is applicable when you collect logs from Docker and Kubernetes clusters.

Scenarios

  • Monitor kernel events, and generate alerts when exceptions occur. The alerts are automatically generated.
  • Collect all system logs for long-term storage to release disk space.
  • Collect the output logs of the software unit for analysis or alerting.
  • Collect all journal logs. This way, you can perform faster log queries and keyword-based queries. These types of queries are more efficient than journalctl-based queries.

Procedure

  1. Log on to the Log Service console.
  2. In the Import Data section, click Custom Data Plug-in.
  3. Select the project and Logstore. Then, click Next.
  4. In the Create Machine Group step, create a machine group.
    • If a machine group is available, click Use Existing Machine Groups.
    • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
      1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Create.

        For more information, see Install Logtail on ECS instances.

        Important If you want to collect logs from an ECS instance that belongs to a different Alibaba Cloud account, a server in an on-premises data center, or a server of a third-party cloud service provider, you must manually install Logtail. For more information, see Install Logtail on a Linux server. After you manually install Logtail, you must configure a user identifier for the server. For more information, see Configure a user identifier.
      2. After Logtail is installed, click Complete Installation.
      3. In the Create Machine Group step, configure the Name parameter and click Next.

        Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom identifier-based machine group.

  5. Select the new machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
    Important If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  6. In the Specify Data Source step, configure the Config Name and Plug-in Config parameters. Then, click Next.
    • inputs specifies the collection configurations of your data source. This parameter is required.
      Important You can specify only one type of data source in the inputs parameter.
    • processors specifies the processing configurations that are used to parse data. You can extract fields, extract log time, desensitize data, and filter logs. This parameter is optional. You can specify one or more processing methods. For more information, see Overview.
    {
      "inputs": [
        {
          "detail": {
            "JournalPaths": [
              "/var/log/journal"
            ],
            "Kernel": true,
            "ParsePriority": true,
            "ParseSyslogFacility": true
          },
          "type": "service_journal"
        }
      ]
    }
    ParameterTypeRequiredDescription
    typestringYesThe type of the data source. Set the value to service_journal.
    JournalPathsString arrayYesThe path to journal logs. We recommend that you set this parameter to the default directory of the journal logs. Example: /var/log/journal.
    SeekPositionstringNoThe method for initial collection. Valid values: head and tail. Default value: tail.
    • If you set this parameter to head, all data is collected.
    • If you set this parameter to tail, only the data that is generated after the Logtail configuration takes effect is collected.
    KernelbooleanNoSpecifies whether to collect kernel logs. Default value: true. This value indicates that kernel logs are collected.
    UnitsString arrayNoThe units whose logs are collected. By default, this parameter is empty, which indicates that the logs of all units are collected.
    ParseSyslogFacilitybooleanNoSpecifies whether to parse the facility field of system logs. Default value: false. If you do not add this parameter, the field is not parsed.
    ParsePrioritybooleanNoSpecifies whether to parse the Priority field. Default value: false. If you do not add this parameter, the field is not parsed.
    If you set this parameter to true, the Priority field is parsed. The following list describes the mapping relationships for the field:
      "0": "emergency"
      "1": "alert"
      "2": "critical"
      "3": "error"
      "4": "warning"
      "5": "notice"
      "6": "informational"
      "7": "debug"
    UseJournalEventTimebooleanNoSpecifies whether to use a field in journal logs as the log time. Default value: false. If you do not add this parameter, the collection time is used as the log time.

    In real-time log collection scenarios, the difference between the log generation time and the log collection time is less than 3 seconds.

  7. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
    Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

Examples

  • Example 1
    Collect journal logs from the default directory /var/log/journal. Logtail configuration:
    {
      "inputs": [
        {
          "detail": {
            "JournalPaths": [
              "/var/log/journal"
            ]
          },
          "type": "service_journal"
        }
      ]
    }
    Sample log:
    MESSAGE:  rejected connection from "192.168.0.250:43936" (error "EOF", ServerName "")
    PACKAGE:  embed
    PRIORITY:  6
    SYSLOG_IDENTIFIER:  etcd
    _BOOT_ID:  fe919cd1268f4721bd87b5c18afe59c3
    _CAP_EFFECTIVE:  0
    _CMDLINE:  /usr/bin/etcd --election-timeout=3000 --heartbeat-interval=500 --snapshot-count=50000 --data-dir=data.etcd --name 192.168.0.251-name-3 --client-cert-auth --trusted-ca-file=/var/lib/etcd/cert/ca.pem --cert-file=/var/lib/etcd/cert/etcd-server.pem --key-file=/var/lib/etcd/cert/etcd-server-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/var/lib/etcd/cert/peer-ca.pem --peer-cert-file=/var/lib/etcd/cert/192.168.0.251-name-3.pem --peer-key-file=/var/lib/etcd/cert/192.168.0.251-name-3-key.pem --initial-advertise-peer-urls https://192.168.0.251:2380 --listen-peer-urls https://192.168.0.251:2380 --advertise-client-urls https://192.168.0.251:2379 --listen-client-urls https://192.168.0.251:2379 --initial-cluster 192.168.0.249-name-1=https://192.168.0.249:2380,192.168.0.250-name-2=https://192.168.0.250:2380,192.168.0.251-name-3=https://192.168.0.251:2380 --initial-cluster-state new --initial-cluster-token abac64c8-baab-4ae6-8412-4253d3cfb0cf
    _COMM:  etcd
    _EXE:  /opt/etcd-v3.3.8/etcd
    _GID:  995
    _HOSTNAME:  iZbp1f7y2ikfe4l8nx95amZ
    _MACHINE_ID:  f0f31005fb5a436d88e3c6cbf54e25aa
    _PID:  10926
    _SOURCE_REALTIME_TIMESTAMP:  1546854068863857
    _SYSTEMD_CGROUP:  /system.slice/etcd.service
    _SYSTEMD_SLICE:  system.slice
    _SYSTEMD_UNIT:  etcd.service
    _TRANSPORT:  journal
    _UID:  997
    __source__:  172.16.1.4
    __tag__:__hostname__:  logtail-ds-8kqb9
    __topic__:  
    _monotonic_timestamp_:  1467135144311
    _realtime_timestamp_:  1546854068864309
  • Example 2
    Collect system logs from hosts in Kubernetes clusters in DaemonSet mode. Use Logtail plug-ins to filter log fields for important fields. Logtail configuration:
    {
      "inputs": [
        {
          "detail": {
            "JournalPaths": [
              "/logtail_host/var/log/journal"
            ],
            "ParsePriority": true,
            "ParseSyslogFacility": true
          },
          "type": "service_journal"
        }
      ],
      "processors": [
        {
          "detail": {
            "Exclude": {
              "UNIT": "^libcontainer.*test"
            }
          },
          "type": "processor_filter_regex"
        },
        {
          "detail": {
            "Include": [
              "MESSAGE",
              "PRIORITY",
              "_EXE",
              "_PID",
              "_SYSTEMD_UNIT",
              "_realtime_timestamp_",
              "_HOSTNAME",
              "UNIT",
              "SYSLOG_FACILITY",
              "SYSLOG_IDENTIFIER"
            ]
          },
          "type": "processor_pick_key"
        }
      ]
    }
    Sample log:
    MESSAGE:  rejected connection from "192.168.0.251:48914" (error "EOF", ServerName "")
    PRIORITY:  informational
    SYSLOG_IDENTIFIER:  etcd
    _EXE:  /opt/etcd-v3.3.8/etcd
    _HOSTNAME:  iZbp1i0czq3zgvxlx7u8ueZ
    _PID:  10590
    _SYSTEMD_UNIT:  etcd.service
    __source__:  172.16.0.141
    __tag__:__hostname__:  logtail-ds-dp48x
    __topic__:  
    _realtime_timestamp_:  1547975837008708

Troubleshooting

If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can troubleshoot the errors based on the instructions that are provided in What do I do if errors occur when I use Logtail to collect logs?

What to do next

After Logtail collects data to Log Service, you can view the data in the Log Service console.