Parameters and configuration examples of two plug-ins that are used to extract log time - Simple Log Service

If you use Logtail to collect logs, you can use the processor_gotime or processor_strptime plug-in to parse the time field in raw logs. This topic describes the parameters of the processor_gotime and processor_strptime plug-ins and provides examples on how to configure the plug-ins.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Time format supported by Go

The processor_gotime plug-in parses the time field in raw logs into the time format supported by Go. You can configure the new time values as the log time for collected logs in Simple Log Service.

Important

Logtail V0.16.28 and later support the processor_gotime plug-in.
Form configuration: You can use form configuration to collect text logs and container stdout and stderr.
Editor configuration in JSON: You cannot use editor configuration in JSON to collect text logs.

Form configuration

Parameters

Set Processor Type to Extract Log Time (Go Time Format).

Then, configure other parameters based on the following table.

Parameter	Description
Original Time Field	The name of the original field.
Original Time Format	The original time format.
Original Time Zone	The original time zone. If you retain System Time Zone, the time zone of the server or container on which Logtail is installed is used.
New Time Field	The name of the new field that is obtained after parsing.
New Time Format	The time format of the new field that is obtained after parsing.
Custom New Time Zone	The time zone of the new field that is obtained after parsing. If you retain System Time Zone, the time zone of the server on which Logtail is installed is used.
Advanced Parameters > Use as Log Time	Specifies whether to use the new time value as the log time.
Advanced Parameters > Retain Original Field	Specifies whether to retain the original field in the new log that is obtained after parsing.
Advanced Parameters > Report Original Field Missing Error	Specifies whether to report an error if the raw log does not contain the original field.
Advanced Parameters > Report Extraction Failure Error	Specifies whether to report an error if the log time fails to be extracted.

Example
The original time field is s_key, and the field value is in the 2006-01-02 15:04:05 (UTC+8) format. The original time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format. The new time value is stored in the d_key field and used as log time in Simple Log Service.
- Raw log
```
"s_key":"2022-07-05 19:28:01"
```
- Logtail plug-in configuration for data processing
- Result
```
"s_key":"2022-07-05 19:28:01"
"d_key":"2022/07/05 20:28:01"
```

Editor configuration in JSON

Parameters

Set type to processor_gotime. Then, configure other parameters in detail based on the following table.

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field.
SourceFormat	String	Yes	The original time format.
SourceLocation	Int	Yes	The original time zone. If the parameter is empty, the time zone of the server or container on which Logtail is installed is used.
DestKey	String	Yes	The name of the new field that is obtained after parsing.
DestFormat	String	Yes	The time format of the new field that is obtained after parsing.
DestLocation	Int	No	The time zone of the new field that is obtained after parsing. If the parameter is empty, the time zone of the server on which Logtail is installed is used.
SetTime	Boolean	No	Specifies whether to use the new time value as the log time. Valid values: true (default) false
KeepSource	Boolean	No	Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values: true (default) false
NoKeyError	Boolean	No	Specifies whether to report an error if the raw log does not contain the original field. Valid values: true (default) false
AlarmIfFail	Boolean	No	Specifies whether to report an error if the log time fails to be extracted. Valid values: true (default) false

Example

The original time field is s_key, and the field value is in the 2006-01-02 15:04:05 (UTC+8) format. The original time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format. The new time value is stored in the d_key field and used as log time in Simple Log Service.

Raw log
```
"s_key":"2019-07-05 19:28:01"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_gotime",
      "detail": {
        "SourceKey": "s_key",
        "SourceFormat":"2006-01-02 15:04:05",
        "SourceLocation":8,
        "DestKey":"d_key",
        "DestFormat":"2006/01/02 15:04:05",
        "DestLocation":9,
        "SetTime": true,
        "KeepSource": true,
        "NoKeyError": true,
        "AlarmIfFail": true
      }
    }
  ]
}

Result

"s_key":"2019-07-05 19:28:01"
"d_key":"2019/07/05 20:28:01"

Time format supported by strptime

The processor_strptime plug-in parses the time field in raw logs into the time format supported by strptime. You can configure the new time values as the log time for collected logs in Simple Log Service. For more information about the time format, see strptime(3) — Linux manual page.

Important

Logtail V0.16.28 and later support the processor_strptime plug-in.

Form configuration

Parameters

Set Processor Type to Extract Log Time (strptime Time Format). Then, configure other parameters based on the following table.

Parameter	Description
Original Field	The name of the original field.
Original Time Format	The original time format.
Retain Original Field	Specifies whether to retain the original field in the new log that is obtained after parsing.
Report Extraction Failure Error	Specifies whether to report an error if the log time fails to be extracted.
Use Time Offset	Specifies whether to configure a time offset.
Time Offset	The offset of the time zone. Unit: seconds. For example, the value 28800 specifies UTC+8, and the value -3600 specifies UTC-1.

Example
The original time field is log_time, and the field value is in the %Y/%m/%d %H:%M:%S format. The original time value is parsed into the log time that uses the time zone of the server on which Logtail is installed. In this example, the time zone is UTC+8.
- Raw log
```
"log_time":"2022/01/02 12:59:59"
```
- Logtail plug-in configuration for data processing
- Result
```
"log_time":"2022/01/02 12:59:59"
Log.Time = 1451710799
```

Common time expressions

Note

The processor_strptime plug-in can parse time values into the %f format. %f is the fractional part of a second. The highest precision that is supported by the processor_strptime plug-in is nanoseconds.

Example	Time expression
2016/01/02 12:59:59	%Y/%m/%d %H:%M:%S
2016/01/02 12:59:59.1	%Y/%m/%d %H:%M:%S.%f
2016/01/02 12:59:59.987654321 +0700 (UTC)	%Y/%m/%d %H:%M:%S.%f %z (%Z)
2016/Jan/02 12:59:59,123456	%Y/%b/%d %H:%M:%S,%f
2019-07-15T04:16:47:123Z	%Y-%m-%dT%H:%M:%S:%f

Editor configuration in JSON

Parameters

Set type to processor_strptime. Then, configure other parameters in detail based on the following table.

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field.
Format	String	Yes	The original time format.
AdjustUTCOffset	Boolean	No	Specifies whether to change the time zone. Valid values: true false (default)
UTCOffset	Int	No	The offset of the time zone. Unit: seconds. For example, the value 28800 specifies UTC+8.
AlarmIfFail	Boolean	No	Specifies whether to report an error if the log time fails to be extracted. Valid values: true (default) false
KeepSource	Boolean	No	Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values: true (default) false
EnablePreciseTimestamp	Boolean	No	Specifies whether to extract time values with high precision. Valid values: true false (default) If you set this parameter to true, the processor_strptime plug-in parses the value of the time field that is specified by SourceKey into a timestamp with millisecond precision and stores the timestamp in the field that is specified by PreciseTimestampKey. Important Before you set this parameter to true, make sure that the value of the time field that is specified by SourceKey uses the required time precision: ms, us, or ns. Only Logtail V1.0.32 and later support this parameter.
PreciseTimestampKey	String	No	The field that stores timestamps with high precision. Default value: precise_timestamp.
PreciseTimestampUnit	String	No	The unit of timestamps with high precision. Default value: ms. Valid values: ms, us, and ns.

Example

The original time field is log_time, and the field value is in the %Y/%m/%d %H:%M:%S format. The original time value is parsed into the log time that uses the time zone of the Logtail server.

Example 1: The time zone is UTC+8.

Raw log
```
"log_time":"2016/01/02 12:59:59"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S"
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59"
Log.Time = 1451710799

Example 2: The time zone is UTC+7.

Raw log
```
"log_time":"2016/01/02 12:59:59"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S",
        "AdjustUTCOffset": true,
        "UTCOffset": 25200
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59"
Log.Time = 1451714399

Example 3: The time zone is UTC +7.

Raw log
```
"log_time":"2016/01/02 12:59:59.123"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S.%f",
        "EnablePreciseTimestamp": true
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59.123"
"precise_timestamp": 1451714399123
Log.Time = 1451714399

Common time expressions

Note

The processor_strptime plug-in can parse time values into the %f format. %f is the fractional part of a second. The highest precision that is supported by the processor_strptime plug-in is nanoseconds.

Example	Time expression
2016/01/02 12:59:59	%Y/%m/%d %H:%M:%S
2016/01/02 12:59:59.1	%Y/%m/%d %H:%M:%S.%f
2016/01/02 12:59:59.987654321 +0700 (UTC)	%Y/%m/%d %H:%M:%S.%f %z (%Z)
2016/Jan/02 12:59:59,123456	%Y/%b/%d %H:%M:%S,%f
2019-07-15T04:16:47:123Z	%Y-%m-%dT%H:%M:%S:%f