how to collect systemd journal logs by using the console - Simple Log Service

Logtail collects Linux systemd journal logs from binary files. This topic describes how to create a Logtail configuration in the Simple Log Service console to collect systemd journal logs.

Prerequisites

Logtail 0.16.18 or later is installed on a Linux server. For more information, see Install Logtail on a Linux server.

Introduction

systemd is a system and service manager for Linux operating systems. When it runs as the init process (PID=1), it acts as the initialization system that starts and maintains user space services. systemd centrally manages the logs of all units, including kernel and application logs. The configuration file is typically /etc/systemd/journald.conf.

Note

The operating system must support the journal log format.

Features

You can set an initial collection position. Checkpoints are automatically saved for subsequent data collection, and the process is not affected by application restarts.
You can filter specified units.
You can collect kernel logs.
Log levels are automatically parsed.
You can collect journal logs from a host within a container. This feature is useful in Docker and Kubernetes scenarios.

Scenarios

Monitor kernel events and automatically trigger alerts for exceptions.
Collect all system logs for long-term storage and reduce disk space usage.
Collect output logs from software (units) for analysis or alerting.
Collect all journal logs to quickly retrieve logs or search for keywords. This significantly improves query efficiency compared to journalctl.

Procedure

Log on to the Simple Log Service console.
In the Data Ingestion area, select Custom Data Plugin.
Select the project and logstore. Then, click Next.
In the Machine Group Configurations step, configure a machine group.
1. Configure the Scenario and Installation Environment parameters as needed.
  Important
  You must configure the Scenario and Installation Environment parameters regardless of whether a machine group is available. The parameter settings affect subsequent configurations.
2. Ensure that a machine group is displayed in the Applied Server Groups section, and click Next.
  Machine group available
  Select a machine group from the Source Machine Group section.
  No machine group available
  Click Create Machine Group. In the Create Machine Group panel, configure the parameters. You can set the Machine Group Identifier parameter to IP Address or Custom Identifier. For more information, see Create a custom identifier-based machine group or Create an IP address-based machine group.
  Important
  If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

On the Data Source Settings tab, set the Configuration Name and Plugin Configuration and then click Next.

inputs is the data source configuration. This parameter is required.
Important
You can configure only one type of data source in an inputs parameter.
processors is the processing configuration used to parse data. This parameter is optional. You can configure one or more processing methods.
If the current inputs configuration does not meet your log parsing requirements, you can add a processors configuration in Plugin Configuration to use Logtail plugins. For example, you can use plugins to extract fields and log time, mask data, and filter logs. For more information, see Use Logtail plugins to process data.

{
  "inputs": [
    {
      "detail": {
        "JournalPaths": [
          "/var/log/journal"
        ],
        "Kernel": true,
        "ParsePriority": true,
        "ParseSyslogFacility": true
      },
      "type": "service_journal"
    }
  ]
}

Parameter	Type	Required	Description
type	string	Yes	The type of the data source. Set this to service_journal.
JournalPaths	string array	Yes	The path of the journal logs. We recommend that you set this to the directory where the journal logs are stored, such as /var/log/journal.
SeekPosition	string	No	The position from which Logtail starts to collect logs for the first time. You can set this to head or tail. The default value is tail. head: Collects all existing data. tail: Collects only new data generated after the Logtail configuration is applied.
Kernel	boolean	No	Specifies whether to collect kernel logs. The default value is true, which indicates that kernel logs are collected.
Units	string array	No	The list of units from which to collect logs. By default, this parameter is empty, which indicates that logs from all units are collected.
ParseSyslogFacility	boolean	No	Specifies whether to parse the facility field of syslog logs. The default value is false. If you do not configure this parameter, the field is not parsed.
ParsePriority	boolean	No	Specifies whether to parse the Priority field. The default value is false. If you do not configure this parameter, the field is not parsed. If you set this to true, the Priority field is parsed based on the following mappings. `"0": "emergency" "1": "alert" "2": "critical" "3": "error" "4": "warning" "5": "notice" "6": "informational" "7": "debug"`
UseJournalEventTime	boolean	No	Specifies whether to use a field in the journal logs as the log time. The default value is false. If you do not configure this parameter, the collection time is used as the log time. For real-time log collection, the time difference is typically within 3 seconds.

Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also manually create field indexes for the collected logs or click Automatic Index Generation. Then, Simple Log Service generates field indexes. For more information, see Create indexes.
Important
If you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.
Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information about how to query and analyze logs, see Guide to log query and analysis.

Examples

Example 1

Collect journal logs from the default /var/log/journal directory. The collection configuration is as follows:

{
  "inputs": [
    {
      "detail": {
        "JournalPaths": [
          "/var/log/journal"
        ]
      },
      "type": "service_journal"
    }
  ]
}

Sample log:

MESSAGE:  rejected connection from "192.168.0.250:43936" (error "EOF", ServerName "")
PACKAGE:  embed
PRIORITY:  6
SYSLOG_IDENTIFIER:  etcd
_BOOT_ID:  fe919cd1268f4721bd87b5c18afe59c3
_CAP_EFFECTIVE:  0
_CMDLINE:  /usr/bin/etcd --election-timeout=3000 --heartbeat-interval=500 --snapshot-count=50000 --data-dir=data.etcd --name 192.168.0.251-name-3 --client-cert-auth --trusted-ca-file=/var/lib/etcd/cert/ca.pem --cert-file=/var/lib/etcd/cert/etcd-server.pem --key-file=/var/lib/etcd/cert/etcd-server-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/var/lib/etcd/cert/peer-ca.pem --peer-cert-file=/var/lib/etcd/cert/192.168.0.251-name-3.pem --peer-key-file=/var/lib/etcd/cert/192.168.0.251-name-3-key.pem --initial-advertise-peer-urls https://192.168.0.251:2380 --listen-peer-urls https://192.168.0.251:2380 --advertise-client-urls https://192.168.0.251:2379 --listen-client-urls https://192.168.0.251:2379 --initial-cluster 192.168.0.249-name-1=https://192.168.0.249:2380,192.168.0.250-name-2=https://192.168.0.250:2380,192.168.0.251-name-3=https://192.168.0.251:2380 --initial-cluster-state new --initial-cluster-token abac64c8-baab-4ae6-8412-4253d3cfb0cf
_COMM:  etcd
_EXE:  /opt/etcd-v3.3.8/etcd
_GID:  995
_HOSTNAME:  iZbp1f7y2ikfe4l8nx95amZ
_MACHINE_ID:  f0f31005fb5a436d88e3c6cbf54e25aa
_PID:  10926
_SOURCE_REALTIME_TIMESTAMP:  1546854068863857
_SYSTEMD_CGROUP:  /system.slice/etcd.service
_SYSTEMD_SLICE:  system.slice
_SYSTEMD_UNIT:  etcd.service
_TRANSPORT:  journal
_UID:  997
__source__:  172.16.1.4
__tag__:__hostname__:  logtail-ds-8kqb9
__topic__:  
_monotonic_timestamp_:  1467135144311
_realtime_timestamp_:  1546854068864309

Example 2

Note

The following limitations apply when you collect journal logs from a host using a container:

If the systemd version in the container is 247 or earlier, you cannot collect journal logs generated by a host with systemd version 249 or later.
If the systemd version in the container is 249 or later, it can read both old and new formats.

In a Kubernetes scenario, you can use the DaemonSet mode to collect system logs from hosts. Because the logs contain many non-critical fields, you can use a processing plug-in to select only the important log fields. The collection configuration is as follows:

{
  "inputs": [
    {
      "detail": {
        "JournalPaths": [
          "/logtail_host/var/log/journal"
        ],
        "ParsePriority": true,
        "ParseSyslogFacility": true
      },
      "type": "service_journal"
    }
  ],
  "processors": [
    {
      "detail": {
        "Exclude": {
          "UNIT": "^libcontainer.*test"
        }
      },
      "type": "processor_filter_regex"
    },
    {
      "detail": {
        "Include": [
          "MESSAGE",
          "PRIORITY",
          "_EXE",
          "_PID",
          "_SYSTEMD_UNIT",
          "_realtime_timestamp_",
          "_HOSTNAME",
          "UNIT",
          "SYSLOG_FACILITY",
          "SYSLOG_IDENTIFIER"
        ]
      },
      "type": "processor_pick_key"
    }
  ]
}

Sample log:

MESSAGE:  rejected connection from "192.168.0.251:48914" (error "EOF", ServerName "")
PRIORITY:  informational
SYSLOG_IDENTIFIER:  etcd
_EXE:  /opt/etcd-v3.3.8/etcd
_HOSTNAME:  iZbp1i0czq3zgvxlx7u8ueZ
_PID:  10590
_SYSTEMD_UNIT:  etcd.service
__source__:  172.16.0.141
__tag__:__hostname__:  logtail-ds-dp48x
__topic__:  
_realtime_timestamp_:  1547975837008708

Troubleshooting

If no data is displayed on the preview page or query page after logs are collected by using Logtail, you can troubleshoot the errors based on the instructions that are provided in What do I do if errors occur when I use Logtail to collect logs?