All Products
Search
Document Center

Simple Log Service:Extract log time

最終更新日:May 21, 2024

If you use Logtail to collect logs, you can use the processor_gotime or processor_strptime plug-in to parse the time field in raw logs. This topic describes the parameters of the processor_gotime and processor_strptime plug-ins and provides examples on how to configure the plug-ins.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Time format supported by Go

The processor_gotime plug-in parses the time field in raw logs into the time format supported by Go. You can configure the new time values as the log time for collected logs in Simple Log Service.

Important
  • Logtail V0.16.28 and later support the processor_gotime plug-in.

  • Form configuration: You can use form configuration to collect text logs and container stdout and stderr.

  • Editor configuration in JSON: You cannot use editor configuration in JSON to collect text logs.

Form configuration

  • Parameters

    Set Processor Type to Extract Log Time (Go Time Format).

    image

    Then, configure other parameters based on the following table.

    Parameter

    Description

    Original Time Field

    The name of the original field.

    Original Time Format

    The original time format.

    Original Time Zone

    The original time zone. If you retain System Time Zone, the time zone of the server or container on which Logtail is installed is used.

    New Time Field

    The name of the new field that is obtained after parsing.

    New Time Format

    The time format of the new field that is obtained after parsing.

    Custom New Time Zone

    The time zone of the new field that is obtained after parsing. If you retain System Time Zone, the time zone of the server on which Logtail is installed is used.

    Advanced Parameters > Use as Log Time

    Specifies whether to use the new time value as the log time.

    Advanced Parameters > Retain Original Field

    Specifies whether to retain the original field in the new log that is obtained after parsing.

    Advanced Parameters > Report Original Field Missing Error

    Specifies whether to report an error if the raw log does not contain the original field.

    Advanced Parameters > Report Extraction Failure Error

    Specifies whether to report an error if the log time fails to be extracted.

  • Example

    The original time field is s_key, and the field value is in the 2006-01-02 15:04:05 (UTC+8) format. The original time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format. The new time value is stored in the d_key field and used as log time in Simple Log Service.

    • Raw log

      "s_key":"2022-07-05 19:28:01"
    • Logtail plug-in configuration for data processing 提取日志时间(Go语言时间格式)

    • Result

      "s_key":"2022-07-05 19:28:01"
      "d_key":"2022/07/05 20:28:01"

Editor configuration in JSON

  • Parameters

    Set type to processor_gotime. Then, configure other parameters in detail based on the following table.

    Parameter

    Type

    Required

    Description

    SourceKey

    String

    Yes

    The name of the original field.

    SourceFormat

    String

    Yes

    The original time format.

    SourceLocation

    Int

    Yes

    The original time zone. If the parameter is empty, the time zone of the server or container on which Logtail is installed is used.

    DestKey

    String

    Yes

    The name of the new field that is obtained after parsing.

    DestFormat

    String

    Yes

    The time format of the new field that is obtained after parsing.

    DestLocation

    Int

    No

    The time zone of the new field that is obtained after parsing. If the parameter is empty, the time zone of the server on which Logtail is installed is used.

    SetTime

    Boolean

    No

    Specifies whether to use the new time value as the log time. Valid values:

    • true (default)

    • false

    KeepSource

    Boolean

    No

    Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values:

    • true (default)

    • false

    NoKeyError

    Boolean

    No

    Specifies whether to report an error if the raw log does not contain the original field. Valid values:

    • true (default)

    • false

    AlarmIfFail

    Boolean

    No

    Specifies whether to report an error if the log time fails to be extracted. Valid values:

    • true (default)

    • false

  • Example

    The original time field is s_key, and the field value is in the 2006-01-02 15:04:05 (UTC+8) format. The original time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format. The new time value is stored in the d_key field and used as log time in Simple Log Service.

    • Raw log

      "s_key":"2019-07-05 19:28:01"
    • Logtail plug-in configuration for data processing

      {
        "processors":[
          {
            "type":"processor_gotime",
            "detail": {
              "SourceKey": "s_key",
              "SourceFormat":"2006-01-02 15:04:05",
              "SourceLocation":8,
              "DestKey":"d_key",
              "DestFormat":"2006/01/02 15:04:05",
              "DestLocation":9,
              "SetTime": true,
              "KeepSource": true,
              "NoKeyError": true,
              "AlarmIfFail": true
            }
          }
        ]
      }
    • Result

      "s_key":"2019-07-05 19:28:01"
      "d_key":"2019/07/05 20:28:01"

Time format supported by strptime

The processor_strptime plug-in parses the time field in raw logs into the time format supported by strptime. You can configure the new time values as the log time for collected logs in Simple Log Service. For more information about the time format, see strptime(3) — Linux manual page.

Important

Logtail V0.16.28 and later support the processor_strptime plug-in.

Form configuration

  • Parameters

    Set Processor Type to Extract Log Time (strptime Time Format). Then, configure other parameters based on the following table.

    Parameter

    Description

    Original Field

    The name of the original field.

    Original Time Format

    The original time format.

    Retain Original Field

    Specifies whether to retain the original field in the new log that is obtained after parsing.

    Report Extraction Failure Error

    Specifies whether to report an error if the log time fails to be extracted.

    Use Time Offset

    Specifies whether to configure a time offset.

    Time Offset

    The offset of the time zone. Unit: seconds. For example, the value 28800 specifies UTC+8, and the value -3600 specifies UTC-1.

  • Example

    The original time field is log_time, and the field value is in the %Y/%m/%d %H:%M:%S format. The original time value is parsed into the log time that uses the time zone of the server on which Logtail is installed. In this example, the time zone is UTC+8.

    • Raw log

      "log_time":"2022/01/02 12:59:59"
    • Logtail plug-in configuration for data processingstrptime时间格式

    • Result

      "log_time":"2022/01/02 12:59:59"
      Log.Time = 1451710799
  • Common time expressions

    Note

    The processor_strptime plug-in can parse time values into the %f format. %f is the fractional part of a second. The highest precision that is supported by the processor_strptime plug-in is nanoseconds.

    Example

    Time expression

    2016/01/02 12:59:59

    %Y/%m/%d %H:%M:%S

    2016/01/02 12:59:59.1

    %Y/%m/%d %H:%M:%S.%f

    2016/01/02 12:59:59.987654321 +0700 (UTC)

    %Y/%m/%d %H:%M:%S.%f %z (%Z)

    2016/Jan/02 12:59:59,123456

    %Y/%b/%d %H:%M:%S,%f

    2019-07-15T04:16:47:123Z

    %Y-%m-%dT%H:%M:%S:%f

Editor configuration in JSON

  • Parameters

    Set type to processor_strptime. Then, configure other parameters in detail based on the following table.

    Parameter

    Type

    Required

    Description

    SourceKey

    String

    Yes

    The name of the original field.

    Format

    String

    Yes

    The original time format.

    AdjustUTCOffset

    Boolean

    No

    Specifies whether to change the time zone. Valid values:

    • true

    • false (default)

    UTCOffset

    Int

    No

    The offset of the time zone. Unit: seconds. For example, the value 28800 specifies UTC+8.

    AlarmIfFail

    Boolean

    No

    Specifies whether to report an error if the log time fails to be extracted. Valid values:

    • true (default)

    • false

    KeepSource

    Boolean

    No

    Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values:

    • true (default)

    • false

    EnablePreciseTimestamp

    Boolean

    No

    Specifies whether to extract time values with high precision. Valid values:

    • true

    • false (default)

    If you set this parameter to true, the processor_strptime plug-in parses the value of the time field that is specified by SourceKey into a timestamp with millisecond precision and stores the timestamp in the field that is specified by PreciseTimestampKey.

    Important
    • Before you set this parameter to true, make sure that the value of the time field that is specified by SourceKey uses the required time precision: ms, us, or ns.

    • Only Logtail V1.0.32 and later support this parameter.

    PreciseTimestampKey

    String

    No

    The field that stores timestamps with high precision. Default value: precise_timestamp.

    PreciseTimestampUnit

    String

    No

    The unit of timestamps with high precision. Default value: ms. Valid values: ms, us, and ns.

  • Example

    The original time field is log_time, and the field value is in the %Y/%m/%d %H:%M:%S format. The original time value is parsed into the log time that uses the time zone of the Logtail server.

    • Example 1: The time zone is UTC+8.

      • Raw log

        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing

        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S"
              }
            }
          ]
        }
      • Result

        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451710799
    • Example 2: The time zone is UTC+7.

      • Raw log

        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing

        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S",
                "AdjustUTCOffset": true,
                "UTCOffset": 25200
              }
            }
          ]
        }
      • Result

        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451714399
    • Example 3: The time zone is UTC +7.

      • Raw log

        "log_time":"2016/01/02 12:59:59.123"
      • Logtail plug-in configuration for data processing

        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S.%f",
                "EnablePreciseTimestamp": true
              }
            }
          ]
        }
      • Result

        "log_time":"2016/01/02 12:59:59.123"
        "precise_timestamp": 1451714399123
        Log.Time = 1451714399
  • Common time expressions

    Note

    The processor_strptime plug-in can parse time values into the %f format. %f is the fractional part of a second. The highest precision that is supported by the processor_strptime plug-in is nanoseconds.

    Example

    Time expression

    2016/01/02 12:59:59

    %Y/%m/%d %H:%M:%S

    2016/01/02 12:59:59.1

    %Y/%m/%d %H:%M:%S.%f

    2016/01/02 12:59:59.987654321 +0700 (UTC)

    %Y/%m/%d %H:%M:%S.%f %z (%Z)

    2016/Jan/02 12:59:59,123456

    %Y/%b/%d %H:%M:%S,%f

    2019-07-15T04:16:47:123Z

    %Y-%m-%dT%H:%M:%S:%f