You can parse the time field in raw logs by using the processor_gotime plug-in or the processor_strptime plug-in. This topic describes the parameters of the processor_gotime plug-in and the processor_strptime plug-in. This topic also provides examples on how to configure the plug-ins.

Time format supported by Go

The processor_gotime plug-in parses the time field in raw logs into the time format supported by Go. You can configure the parsed time value as log time in Log Service. For more information, see Go.

  • Parameters
    The following table describes the parameters that you can configure in detail if you set the type parameter to processor_gotime.
    Note Only Logtail V0.16.28 and later support the processor_gotime plug-in.
    Parameter Type Required Description
    SourceKey String Yes The name of the raw field.
    SourceFormat String Yes The raw time format.
    SourceLocation Int Yes The raw time zone. If the parameter is empty, the time zone of the server or container on which Logtail runs is used.
    DestKey String Yes The name of the parsed field.
    DestFormat String Yes The parsed time format.
    DestLocation Int No The parsed time zone. If the parameter is empty, the time zone of the server on which Logtail runs is used.
    SetTime Boolean No Specifies whether to configure the parsed time value as log time. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that the parsed time value is configured as log time.
    KeepSource Boolean No Specifies whether to retain the raw field. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that the raw field is retained.
    NoKeyError Boolean No Specifies whether to report an error if the raw field is not found. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that an error is reported if the raw field is not found.
    AlarmIfFail Boolean No Specifies whether to trigger an alert if log time fails to be extracted. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that an alert is triggered if log time fails to be extracted.
  • Example
    In this example, the raw time value in the 2006-01-02 15:04:05 (UTC+8) format is extracted from the s_key field and parsed into the format 2006/01/02 15:04:05 (UTC+9). The parsed time value is configured as log time in Log Service and stored in the d_key field.
    • Raw log
      "s_key":"2019-07-05 19:28:01"
    • Logtail plug-in configuration for data processing
      {
        "processors":[
          {
            "type":"processor_gotime",
            "detail": {
              "SourceKey": "s_key",
              "SourceFormat":"2006-01-02 15:04:05",
              "SourceLocation":8,
              "DestKey":"d_key",
              "DestFormat":"2006/01/02 15:04:05",
              "DestLocation":9,
              "SetTime": true,
              "KeepSource": true,
              "NoKeyError": true,
              "AlarmIfFail": true
            }
          }
        ]
      }
    • Results
      "s_key":"2019-07-05 19:28:01"
      "d_key":"2019/07/05 20:28:01"

Time format supported by strptime

The processor_strptime plug-in parses the time field in raw logs into the time format supported by strptime. You can configure the parsed time value as log time in Log Service. For more information, see strptime.

  • Parameters
    The following table describes the parameters that you can configure in detail if you set the type parameter to processor_strptime.
    Note Only Logtail V0.16.28 and later support the processor_strptime plug-in.
    Parameter Type Required Description
    SourceKey String Yes The name of the raw field.
    Format String Yes The raw time format.
    AdjustUTCOffset Boolean No Specifies whether to change the time zone. If you do not include this parameter in the configuration, the system uses the value false by default. The value false indicates that the time zone is not changed.
    UTCOffset Int No The offset that you want to use to change the time zone. Unit: seconds. For example, the value 28800 indicates that the time zone is changed to UTC+8.
    AlarmIfFail Boolean No Specifies whether to trigger an alert if log time fails to be extracted. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that an alert is triggered if log time fails to be extracted.
    KeepSource Boolean No Specifies whether to retain the raw field. If you do not include this parameter in the configuration, the system uses the value true by default. The value true indicates that the raw field is retained.
    EnablePreciseTimestamp Boolean No Specifies whether to extract time values with high precision. If you do not include this parameter in the configuration, the system uses the value false by default. The value false indicates that time values with high precision are not extracted.
    If you set this parameter to true, the processor_strptime plug-in parses the value of the SourceKey field into a timestamp with millisecond precision and stores the timestamp in the field specified by the PreciseTimestampKey parameter.
    Note
    • Before you set this parameter to true, make sure that the value of the SourceKey field uses one of the following time precisions: ms, us, and ns.
    • Only Logtail V1.0.32 and later support this parameter.
    PreciseTimestampKey String No The field that stores timestamps with high precision. If you do not include this parameter in the configuration, the system uses the precise_timestamp field by default.
    PreciseTimestampUnit String No The unit of the timestamp with high precision. If you do not include this parameter in the configuration, the system uses ms by default. Valid values: ms, us, and ns.
  • Examples
    In the following examples, the time value in the %Y/%m/%d %H:%M:%S format is extracted from the log_time field and parsed into the log time in the format that you specify. The time zone of your server is used.
    • Example 1: The time zone is UTC+8.
      • Raw log
        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S"
              }
            }
          ]
        }
      • Results
        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451710799
    • Example 2: The time zone is UTC+7.
      • Raw log
        "log_time":"2016/01/02 12:59:59"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S",
                "AdjustUTCOffset": true,
                "UTCOffset": 25200
              }
            }
          ]
        }
      • Results
        "log_time":"2016/01/02 12:59:59"
        Log.Time = 1451714399
    • Example 3: The time zone is UTC +7.
      • Raw log
        "log_time":"2016/01/02 12:59:59.123"
      • Logtail plug-in configuration for data processing
        {
          "processors":[
            {
              "type":"processor_strptime",
              "detail": {
                "SourceKey": "log_time",
                "Format": "%Y/%m/%d %H:%M:%S.%f",
                "EnablePreciseTimestamp": true
              }
            }
          ]
        }
      • Results
        "log_time":"2016/01/02 12:59:59.123"
        "precise_timestamp": 1451714399123
        Log.Time = 1451714399
  • Common time expressions
    Note The processor_strptime plug-in can parse time values in the %f format. %f indicates the fractional part of the second. The highest precision that is supported by the processor_strptime plug-in is the nanosecond.
    Example Time expression
    2016/01/02 12:59:59 %Y/%m/%d %H:%M:%S
    2016/01/02 12:59:59.1 %Y/%m/%d %H:%M:%S.%f
    2016/01/02 12:59:59.987654321 +0700 (UTC) %Y/%m/%d %H:%M:%S.%f %z (%Z)
    2016/Jan/02 12:59:59,123456 %Y/%b/%d %H:%M:%S,%f
    2019-07-15T04:16:47:123Z %Y-%m-%dT%H:%M:%S:%f