Collect systemd-journald logs

Overview

systemd is a system and service manager for Linux. When systemd runs as an initialization process (PID = 1), it boots and maintains the services of user spaces. systemd manages kernel and application logs in a centralized manner. The configuration file that is used to manage these logs is /etc/systemd/journald.conf.

Features

• Allows you to set the initial collection position. Checkpoints are automatically saved for subsequent data collection. The process is not affected when applications are restarted.
• Allows you to filter specified applications.
• Allows you to collect kernel logs.
• Supports automatic parsing of log severity.
• Allows you to run systemd as a container to collect journal logs from hosts. This feature is applicable when you collect logs from Docker and Kubernetes clusters.

Scenarios

• Monitor kernel events, and trigger alerts if exceptions occur.
• Collect system logs for long-term storage and release disk space.
• Collect application logs for analysis or alerts.
• Collect all journal logs and query log data with higher efficiency than journalctl.

Procedure

1. Log on to the Log Service console.
2. In the Import Data section, select Custom Data Plug-in.
3. In the Specify Logstore step, select the target project and Logstore, and click Next.
   You can also click Create Now to create a project and a Logstore. For more information, see Step 1: Create a project and a Logstore.
4. In the Create Machine Group step, create a machine group.
   • If a machine group is available, click Using Existing Machine Groups.
   • If no machine group is available, perform the following steps. The following steps take ECS instances as an example to describe how to create a machine group:
     1. Install Logtail on ECS instances. For more information, see Install Logtail on ECS instances.

        If Logtail is installed on the ECS instances, click Complete Installation.

        Note If you want to collect logs from a user-created cluster, you must manually install Logtail on the servers in the cluster. For more information, see Install Logtail in Linux.
     2. After the installation is completed, click Complete Installation.
     3. On the page that appears, set relevant parameters for the machine group. For more information, see Create an IP address-based machine group or Create a custom ID-based machine group.
5. In the Machine Group Settings step, apply the configurations to the machine group.
   Select the created machine group and move the group from Source Server Groups to Applied Server Groups.
6. In the Specify Data Source step, set the Config Name and Plug-in Config parameters.
   • inputs: Required. The Logtail configurations for log collection.
     Note You can configure only one type of data source in the inputs field.
   • processors: Optional. The Logtail configurations for data processing. You can configure one or more processing methods in the processors field. For more information, see Process data.

   {
     "inputs": [
       {
         "detail": {
           "JournalPaths": [
             "/var/log/journal"
           ]
         },
         "type": "service_journal"
       }
     ]
   }

   Parameter	Type	Required	Description
   type	String	Yes	The type of the data source. Set the value to service_journal.
   JournalPaths	String array	Yes	The path of journal logs. We recommend that you set the value to the directory of journal logs, for example, /var/log/journal.
   SeekPosition	String	No	The method of the first collection. Valid values: head and tail. Default value: tail. If you set the value to head, all data is collected. If you set the value to tail, only the data that is generated after the Logtail configurations take effect is collected.
   Kernel	Boolean	No	Specifies whether to collect kernel logs. Default value: true. This value indicates that kernel logs are collected.
   Units	String array	No	The applications from which logs are collected. Default value: null. This value indicates that logs from all applications are collected.
   ParseSyslogFacility	Boolean	No	Specifies whether to parse the facility field of syslog logs. Default value: false. This value indicates that the facility field of syslog logs is not parsed.
   ParsePriority	Boolean	No	Specifies whether to parse the priority field of syslog logs. Default value: false. This value indicates that the priority field of syslog logs is not parsed. If you set the value to true, the priority field of syslog logs is parsed based on the following mapping relationships: "0": "emergency" "1": "alert" "2": "critical" "3": "error" "4": "warning" "5": "notice" "6": "informational" "7": "debug"
   UseJournalEventTime	Boolean	No	Specifies whether to use the field of journal logs as the timestamp of the log. Default value: false. This value indicates that the time when a log entry is collected is recorded as the timestamp of the log entry. In most cases, the difference between the log collection time and the log timestamp is less than 3 seconds.

7. In the Configure Query and Analysis step, configure the indexes.
   Indexes are configured by default. You can re-configure the indexes based on your business requirements. For more information, see Enable and configure the index feature for a Logstore.

   Note

   • You must configure Full Text Index or Field Search. If you configure both of them, the settings of Field Search are applied.
   • If the data type of index is long or double, the Case Sensitive and Delimiter settings are unavailable.

Examples

• Example 1
  To collect journal logs from the /var/log/journal directory, you can use the following configuration file:

  {
    "inputs": [
      {
        "detail": {
          "JournalPaths": [
            "/var/log/journal"
          ]
        },
        "type": "service_journal"
      }
    ]
  }

  Sample log entry:

  MESSAGE:  rejected connection from "192.168.0.250:43936" (error "EOF", ServerName "")
  PACKAGE:  embed
  PRIORITY:  6
  SYSLOG_IDENTIFIER:  etcd
  _BOOT_ID:  fe919cd1268f4721bd87b5c18afe59c3
  _CAP_EFFECTIVE:  0
  _CMDLINE:  /usr/bin/etcd --election-timeout=3000 --heartbeat-interval=500 --snapshot-count=50000 --data-dir=data.etcd --name 192.168.0.251-name-3 --client-cert-auth --trusted-ca-file=/var/lib/etcd/cert/ca.pem --cert-file=/var/lib/etcd/cert/etcd-server.pem --key-file=/var/lib/etcd/cert/etcd-server-key.pem --peer-client-cert-auth --peer-trusted-ca-file=/var/lib/etcd/cert/peer-ca.pem --peer-cert-file=/var/lib/etcd/cert/192.168.0.251-name-3.pem --peer-key-file=/var/lib/etcd/cert/192.168.0.251-name-3-key.pem --initial-advertise-peer-urls https://192.168.0.251:2380 --listen-peer-urls https://192.168.0.251:2380 --advertise-client-urls https://192.168.0.251:2379 --listen-client-urls https://192.168.0.251:2379 --initial-cluster 192.168.0.249-name-1=https://192.168.0.249:2380,192.168.0.250-name-2=https://192.168.0.250:2380,192.168.0.251-name-3=https://192.168.0.251:2380 --initial-cluster-state new --initial-cluster-token abac64c8-baab-4ae6-8412-4253d3cfb0cf
  _COMM:  etcd
  _EXE:  /opt/etcd-v3.3.8/etcd
  _GID:  995
  _HOSTNAME:  iZbp1f7y2ikfe4l8nx95amZ
  _MACHINE_ID:  f0f31005fb5a436d88e3c6cbf54e25aa
  _PID:  10926
  _SOURCE_REALTIME_TIMESTAMP:  1546854068863857
  _SYSTEMD_CGROUP:  /system.slice/etcd.service
  _SYSTEMD_SLICE:  system.slice
  _SYSTEMD_UNIT:  etcd.service
  _TRANSPORT:  journal
  _UID:  997
  __source__:  172.16.1.4
  __tag__:__hostname__:  logtail-ds-8kqb9
  __topic__:  
  _monotonic_timestamp_:  1467135144311
  _realtime_timestamp_:  1546854068864309

• Example 2
  To collect important fields from system logs of Kubernetes hosts in the DaemonSet mode, you can use the following configuration file:

  {
    "inputs": [
      {
        "detail": {
          "JournalPaths": [
            "/logtail_host/var/log/journal"
          ],
          "ParsePriority": true,
          "ParseSyslogFacility": true
        },
        "type": "service_journal"
      }
    ],
    "processors": [
      {
        "detail": {
          "Exclude": {
            "UNIT": "^libcontainer.*test"
          }
        },
        "type": "processor_filter_regex"
      },
      {
        "detail": {
          "Include": [
            "MESSAGE",
            "PRIORITY",
            "_EXE",
            "_PID",
            "_SYSTEMD_UNIT",
            "_realtime_timestamp_",
            "_HOSTNAME",
            "UNIT",
            "SYSLOG_FACILITY",
            "SYSLOG_IDENTIFIER"
          ]
        },
        "type": "processor_pick_key"
      }
    ]
  }

  Sample log entry:

  MESSAGE:  rejected connection from "192.168.0.251:48914" (error "EOF", ServerName "")
  PRIORITY:  informational
  SYSLOG_IDENTIFIER:  etcd
  _EXE:  /opt/etcd-v3.3.8/etcd
  _HOSTNAME:  iZbp1i0czq3zgvxlx7u8ueZ
  _PID:  10590
  _SYSTEMD_UNIT:  etcd.service
  __source__:  172.16.0.141
  __tag__:__hostname__:  logtail-ds-dp48x
  __topic__:  
  _realtime_timestamp_:  1547975837008708