When you use Logtail to collect logs, you can parse the time field in raw logs by using the processor_gotime or processor_strptime plug-in. This topic describes the parameters of the processor_gotime and processor_strptime plug-ins. This topic also provides examples on how to configure the plug-ins.

Important You can use form configuration to add Logtail plug-ins only if you select Kubernetes - Standard Output in the Import Data section.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Entry point.

Time format supported by Go

The processor_gotime plug-in parses the time field in raw logs into the time format supported by Go. You can configure the new time values as the log time for collected logs in Log Service. For more information, see Go.

Important Logtail V0.16.28 and later support the processor_gotime plug-in.

Form configuration

  • Parameters
    Set Processor Type to Extract Log Time (Go Time Format). Then, configure other parameters based on the following table.
    ParameterDescription
    Original Time FieldThe name of the original field.
    Original Time FormatThe original time format.
    Original Time ZoneThe original time zone. If you select System Timezone, the time zone of the server or container on which Logtail runs is used.
    New Time FieldThe name of the new field that is obtained after parsing.
    New Time FormatThe new time format that is obtained after parsing.
    Custom New Time ZoneThe new time zone that is obtained after parsing. If you select System Timezone, the time zone of the server on which Logtail runs is used.
    Use as Log TimeSpecifies whether to use the new time value as the log time.
    Retain Original FieldSpecifies whether to retain the original field in the new log that is obtained after parsing.
    Report Original Field Missing ErrorSpecifies whether to report an error if the raw log does not contain the original field.
    Report Extraction Failure ErrorSpecifies whether to report an error if the log time fails to be extracted.
  • Example
    The time field in the raw log is s_key, and the value of the time field is in the 2006-01-02 15:04:05 (UTC+8) format. The raw time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format, and the new time value is stored in the d_key field and used as the log time for collected logs in Log Service.
    • Raw log
      "s_key":"2022-07-05 19:28:01"
    • Logtail plug-in configuration for data processingExtract Log Time (Go Time Format)
    • Result
      "s_key":"2022-07-05 19:28:01"
      "d_key":"2022/07/05 20:28:01"

Editor configuration in JSON

  • Parameters
    Set type to processor_gotime. Then, configure other parameters in detail based on the following table.
    ParameterTypeRequiredDescription
    SourceKeyStringYesThe name of the original field.
    SourceFormatStringYesThe original time format.
    SourceLocationIntYesThe original time zone. If the parameter is empty, the time zone of the server or container on which Logtail runs is used.
    DestKeyStringYesThe name of the new field that is obtained after parsing.
    DestFormatStringYesThe new time format that is obtained after parsing.
    DestLocationIntNoThe new time zone that is obtained after parsing. If the parameter is empty, the time zone of the server on which Logtail runs is used.
    SetTimeBooleanNoSpecifies whether to use the new time value as the log time. Valid values:
    • true: uses the new time value as the log time. This is the default value.
    • false: does not use the new time value as the log time.
    KeepSourceBooleanNoSpecifies whether to retain the original field in the new log that is obtained after parsing. Valid values:
    • true: retains the original field. This is the default value.
    • false: does not retain the original field.
    NoKeyErrorBooleanNoSpecifies whether to report an error if the raw log does not contain the original field. Valid values:
    • true: reports an error. This is the default value.
    • false: does not report an error.
    AlarmIfFailBooleanNoSpecifies whether to report an error if the log time fails to be extracted. Valid values:
    • true: reports an error. This is the default value.
    • false: does not report an error.
  • Example
    The time field in the raw log is s_key, and the value of the time field is in the 2006-01-02 15:04:05 (UTC+8) format. The raw time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format, and the new time value is stored in the d_key field and used as the log time for collected logs in Log Service.
    • Raw log
      "s_key":"2019-07-05 19:28:01"
    • Logtail plug-in configuration for data processing
      {
        "processors":[
          {
            "type":"processor_gotime",
            "detail": {
              "SourceKey": "s_key",
              "SourceFormat":"2006-01-02 15:04:05",
              "SourceLocation":8,
              "DestKey":"d_key",
              "DestFormat":"2006/01/02 15:04:05",
              "DestLocation":9,
              "SetTime": true,
              "KeepSource": true,
              "NoKeyError": true,
              "AlarmIfFail": true
            }
          }
        ]
      }
    • Result
      "s_key":"2019-07-05 19:28:01"
      "d_key":"2019/07/05 20:28:01"

Time format supported by strptime

The processor_strptime plug-in parses the time field in raw logs into the time format supported by strptime. You can configure the new time values as the log time for collected logs in Log Service. For more information, see strptime.

Important Logtail V0.16.28 and later support the processor_strptime plug-in.

Form configuration

  • Parameters
    Set Processor Type to Extract Log Time (strptime Time Format). Then, configure other parameters based on the following table.
    ParameterDescription
    Original FieldThe name of the original field.
    Original Time FormatThe original time format.
    Retain Original FieldSpecifies whether to retain the original field in the new log that is obtained after parsing.
    Report Extraction Failure ErrorSpecifies whether to report an error if the log time fails to be extracted.
    Use Time OffsetSpecifies whether to configure the time offset.
    Time OffsetThe offset that you want to use to change the time zone. Unit: seconds. For example, the value 28800 indicates that the time zone is changed to UTC+8.
    Extract High-precision TimeSpecifies whether to extract time values with high precision. If you select this parameter, the processor_strptime plug-in parses the value of the time field that is specified by the Original Field parameter into a timestamp with millisecond precision and stores the timestamp in the field that is specified by the High-precision Time Field Name parameter.
    Important
    • Before you select this parameter, make sure that the value of the time field that is specified by the Original Field parameter uses one of the following time precisions: ms, us, and ns.
    • Only Logtail V1.0.32 and later support this parameter.
    High-precision Time Field NameThe field that stores timestamps with high precision. Default value: precise_timestamp.
    Time UnitThe unit of timestamps with high precision. Valid values: ms, us, and ns.
  • Configuration example
    The time field in the raw log is log_time, and the value of the time field is in the %Y/%m/%d %H:%M:%S format. The raw time value is parsed into the log time in the format that you specify. The time zone of your server is used. The time zone is UTC+8.
    • Raw log
      "log_time":"2022/01/02 12:59:59"
    • Logtail plug-in configuration for data processing Time format supported by strptime
    • Result
      "log_time":"2022/01/02 12:59:59"
      Log.Time = 1451710799
  • Common time expressions
    Note The processor_strptime plug-in can parse time values into the %f format. %f indicates the fractional part of the second. The highest precision that is supported by the processor_strptime plug-in is the nanosecond.
    ExampleTime expression
    2016/01/02 12:59:59%Y/%m/%d %H:%M:%S
    2016/01/02 12:59:59.1%Y/%m/%d %H:%M:%S.%f
    2016/01/02 12:59:59.987654321 +0700 (UTC)%Y/%m/%d %H:%M:%S.%f %z (%Z)
    2016/Jan/02 12:59:59,123456%Y/%b/%d %H:%M:%S,%f
    2019-07-15T04:16:47:123Z%Y-%m-%dT%H:%M:%S:%f

Editor configuration in JSON

  • Parameters
    Set type to processor_strptime. Then, configure other parameters in detail based on the following table.
    ParameterTypeRequiredDescription
    SourceKeyStringYesThe name of the original field.
    FormatStringYesThe original time format.
    AdjustUTCOffsetBooleanNoSpecifies whether to change the time zone. Valid values:
    • true: changes the time zone.
    • false: does not change the time zone. This is the default value.
    UTCOffsetIntNoThe offset that you want to use to change the time zone. Unit: seconds. For example, the value 28800 indicates that the time zone is changed to UTC+8.
    AlarmIfFailBooleanNoSpecifies whether to report an error if the log time fails to be extracted. Valid values:
    • true: reports an error. This is the default value.
    • false: does not report an error.
    KeepSourceBooleanNoSpecifies whether to retain the original field in the new log that is obtained after parsing. Valid values:
    • true: retains the original field. This is the default value.
    • false: does not retain the original field.
    EnablePreciseTimestampBooleanNoSpecifies whether to extract time values with high precision. Valid values:
    • true: extracts time values with high precision.
    • false: does not extract time values with high precision. This is the default value.
    If you set this parameter to true, the processor_strptime plug-in parses the value of the time field that is specified by the SourceKey parameter into a timestamp with millisecond precision and stores the timestamp in the field that is specified by the PreciseTimestampKey parameter.
    Important
    • Before you set this parameter to true, make sure that the value of the time field that is specified by the SourceKey parameter uses one of the following time precisions: ms, us, and ns.
    • Only Logtail V1.0.32 and later support this parameter.
    PreciseTimestampKeyStringNoThe field that stores timestamps with high precision. Default value: precise_timestamp.
    PreciseTimestampUnitStringNoThe unit of timestamps with high precision. Default value: ms. Valid values: ms, us, and ns.
  • Examples
    The time field in the raw log is log_time, and the value of the time field is in the %Y/%m/%d %H:%M:%S format. The raw time value is parsed into the log time in the format that you specify. The time zone of your server is used.
    • Example 1: The time zone is UTC+8.
      • Raw log
        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S"
              }
            }
          ]
        }
      • Result
        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451710799
    • Example 2: The time zone is UTC+7.
      • Raw log
        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S",
                "AdjustUTCOffset": true,
                "UTCOffset": 25200
              }
            }
          ]
        }
      • Result
        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451714399
    • Example 3: The time zone is UTC +7.
      • Raw log
        "log_time":"2016/01/02 12:59:59.123"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S.%f",
                "EnablePreciseTimestamp": true
              }
            }
          ]
        }
      • Result
        "log_time":"2016/01/02 12:59:59.123"
        "precise_timestamp": 1451714399123
        Log.Time = 1451714399
  • Common time expressions
    Note The processor_strptime plug-in can parse time values into the %f format. %f indicates the fractional part of the second. The highest precision that is supported by the processor_strptime plug-in is the nanosecond.
    ExampleTime expression
    2016/01/02 12:59:59%Y/%m/%d %H:%M:%S
    2016/01/02 12:59:59.1%Y/%m/%d %H:%M:%S.%f
    2016/01/02 12:59:59.987654321 +0700 (UTC)%Y/%m/%d %H:%M:%S.%f %z (%Z)
    2016/Jan/02 12:59:59,123456%Y/%b/%d %H:%M:%S,%f
    2019-07-15T04:16:47:123Z%Y-%m-%dT%H:%M:%S:%f