Extract log time - Simple Log Service - Alibaba Cloud Documentation Center

When you use Logtail to collect logs, you can parse the time field in raw logs by using the processor_gotime or processor_strptime plug-in. This topic describes the parameters of the processor_gotime and processor_strptime plug-ins. This topic also provides examples on how to configure the plug-ins.

Important

Form configuration: You can use form configuration to collect text logs and container stdout and stderr.
Editor configuration in JSON: You cannot use editor configuration in JSON to collect text logs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Time format supported by Go

The processor_gotime plug-in parses the time field in raw logs into the time format supported by Go. You can configure the new time values as the log time for collected logs in Log Service.

Important

Logtail V0.16.28 and later support the processor_gotime plug-in.

Form configuration

Parameters

Set Processor Type to Extract Log Time (Go Time Format). Then, configure other parameters based on the following table.

Parameter	Description
Original Time Field	The name of the original field.
Original Time Format	The original time format.
Original Time Zone	The original time zone. If you select System Timezone, the time zone of the server or container on which Logtail runs is used.
New Time Field	The name of the new field that is obtained after parsing.
New Time Format	The new time format that is obtained after parsing.
Custom New Time Zone	The new time zone that is obtained after parsing. If you select System Timezone, the time zone of the server on which Logtail runs is used.
Use as Log Time	Specifies whether to use the new time value as the log time.
Retain Original Field	Specifies whether to retain the original field in the new log that is obtained after parsing.
Report Original Field Missing Error	Specifies whether to report an error if the raw log does not contain the original field.
Report Extraction Failure Error	Specifies whether to report an error if the log time fails to be extracted.

Example
The time field in the raw log is s_key, and the value of the time field is in the 2006-01-02 15:04:05 (UTC+8) format. The raw time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format, and the new time value is stored in the d_key field and used as the log time for collected logs in Log Service.
- Raw log
```
"s_key":"2022-07-05 19:28:01"
```
- Logtail plug-in configuration for data processing
- Result
```
"s_key":"2022-07-05 19:28:01"
"d_key":"2022/07/05 20:28:01"
```

Editor configuration in JSON

Parameters

Set type to processor_gotime. Then, configure other parameters in detail based on the following table.

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field.
SourceFormat	String	Yes	The original time format.
SourceLocation	Int	Yes	The original time zone. If the parameter is empty, the time zone of the server or container on which Logtail runs is used.
DestKey	String	Yes	The name of the new field that is obtained after parsing.
DestFormat	String	Yes	The new time format that is obtained after parsing.
DestLocation	Int	No	The new time zone that is obtained after parsing. If the parameter is empty, the time zone of the server on which Logtail runs is used.
SetTime	Boolean	No	Specifies whether to use the new time value as the log time. Valid values: true: uses the new time value as the log time. This is the default value. false: does not use the new time value as the log time.
KeepSource	Boolean	No	Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values: true: retains the original field. This is the default value. false: does not retain the original field.
NoKeyError	Boolean	No	Specifies whether to report an error if the raw log does not contain the original field. Valid values: true: reports an error. This is the default value. false: does not report an error.
AlarmIfFail	Boolean	No	Specifies whether to report an error if the log time fails to be extracted. Valid values: true: reports an error. This is the default value. false: does not report an error.

Example

The time field in the raw log is s_key, and the value of the time field is in the 2006-01-02 15:04:05 (UTC+8) format. The raw time value is parsed into the 2006/01/02 15:04:05 (UTC+9) format, and the new time value is stored in the d_key field and used as the log time for collected logs in Log Service.

Raw log
```
"s_key":"2019-07-05 19:28:01"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_gotime",
      "detail": {
        "SourceKey": "s_key",
        "SourceFormat":"2006-01-02 15:04:05",
        "SourceLocation":8,
        "DestKey":"d_key",
        "DestFormat":"2006/01/02 15:04:05",
        "DestLocation":9,
        "SetTime": true,
        "KeepSource": true,
        "NoKeyError": true,
        "AlarmIfFail": true
      }
    }
  ]
}

Result

"s_key":"2019-07-05 19:28:01"
"d_key":"2019/07/05 20:28:01"

Time format supported by strptime

The processor_strptime plug-in parses the time field in raw logs into the time format supported by strptime. You can configure the new time values as the log time for collected logs in Log Service. For more information, see strptime.

Important

Logtail V0.16.28 and later support the processor_strptime plug-in.

Form configuration

Parameters

Set Processor Type to Extract Log Time (strptime Time Format). Then, configure other parameters based on the following table.

Parameter	Description
Original Field	The name of the original field.
Original Time Format	The original time format.
Retain Original Field	Specifies whether to retain the original field in the new log that is obtained after parsing.
Report Extraction Failure Error	Specifies whether to report an error if the log time fails to be extracted.
Use Time Offset	Specifies whether to configure the time offset.
Time Offset	The offset that you want to use to change the time zone. Unit: seconds. For example, the value 28800 indicates that the time zone is changed to UTC+8.
Extract High-precision Time	Specifies whether to extract time values with high precision. If you select this parameter, the processor_strptime plug-in parses the value of the time field that is specified by the Original Field parameter into a timestamp with millisecond precision and stores the timestamp in the field that is specified by the High-precision Time Field Name parameter. Important Before you select this parameter, make sure that the value of the time field that is specified by the Original Field parameter uses one of the following time precisions: ms, us, and ns. Only Logtail V1.0.32 and later support this parameter.
High-precision Time Field Name	The field that stores timestamps with high precision. Default value: precise_timestamp.
Time Unit	The unit of timestamps with high precision. Valid values: ms, us, and ns.

Configuration example
The time field in the raw log is log_time, and the value of the time field is in the %Y/%m/%d %H:%M:%S format. The raw time value is parsed into the log time in the format that you specify. The time zone of your server is used. The time zone is UTC+8.
- Raw log
```
"log_time":"2022/01/02 12:59:59"
```
- Logtail plug-in configuration for data processing
- Result
```
"log_time":"2022/01/02 12:59:59"
Log.Time = 1451710799
```

Common time expressions

Note

The processor_strptime plug-in can parse time values into the %f format. %f indicates the fractional part of the second. The highest precision that is supported by the processor_strptime plug-in is the nanosecond.

Example	Time expression
2016/01/02 12:59:59	%Y/%m/%d %H:%M:%S
2016/01/02 12:59:59.1	%Y/%m/%d %H:%M:%S.%f
2016/01/02 12:59:59.987654321 +0700 (UTC)	%Y/%m/%d %H:%M:%S.%f %z (%Z)
2016/Jan/02 12:59:59,123456	%Y/%b/%d %H:%M:%S,%f
2019-07-15T04:16:47:123Z	%Y-%m-%dT%H:%M:%S:%f

Editor configuration in JSON

Parameters

Set type to processor_strptime. Then, configure other parameters in detail based on the following table.

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the original field.
Format	String	Yes	The original time format.
AdjustUTCOffset	Boolean	No	Specifies whether to change the time zone. Valid values: true: changes the time zone. false: does not change the time zone. This is the default value.
UTCOffset	Int	No	The offset that you want to use to change the time zone. Unit: seconds. For example, the value 28800 indicates that the time zone is changed to UTC+8.
AlarmIfFail	Boolean	No	Specifies whether to report an error if the log time fails to be extracted. Valid values: true: reports an error. This is the default value. false: does not report an error.
KeepSource	Boolean	No	Specifies whether to retain the original field in the new log that is obtained after parsing. Valid values: true: retains the original field. This is the default value. false: does not retain the original field.
EnablePreciseTimestamp	Boolean	No	Specifies whether to extract time values with high precision. Valid values: true: extracts time values with high precision. false: does not extract time values with high precision. This is the default value. If you set this parameter to true, the processor_strptime plug-in parses the value of the time field that is specified by the SourceKey parameter into a timestamp with millisecond precision and stores the timestamp in the field that is specified by the PreciseTimestampKey parameter. Important Before you set this parameter to true, make sure that the value of the time field that is specified by the SourceKey parameter uses one of the following time precisions: ms, us, and ns. Only Logtail V1.0.32 and later support this parameter.
PreciseTimestampKey	String	No	The field that stores timestamps with high precision. Default value: precise_timestamp.
PreciseTimestampUnit	String	No	The unit of timestamps with high precision. Default value: ms. Valid values: ms, us, and ns.

Examples

The time field in the raw log is log_time, and the value of the time field is in the %Y/%m/%d %H:%M:%S format. The raw time value is parsed into the log time in the format that you specify. The time zone of your server is used.

Example 1: The time zone is UTC+8.

Raw log
```
"log_time":"2016/01/02 12:59:59"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S"
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59"
Log.Time = 1451710799

Example 2: The time zone is UTC+7.

Raw log
```
"log_time":"2016/01/02 12:59:59"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S",
        "AdjustUTCOffset": true,
        "UTCOffset": 25200
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59"
Log.Time = 1451714399

Example 3: The time zone is UTC +7.

Raw log
```
"log_time":"2016/01/02 12:59:59.123"
```

Logtail plug-in configuration for data processing

{
  "processors":[
    {
      "type":"processor_strptime",
      "detail": {
        "SourceKey": "log_time",
        "Format": "%Y/%m/%d %H:%M:%S.%f",
        "EnablePreciseTimestamp": true
      }
    }
  ]
}

Result

"log_time":"2016/01/02 12:59:59.123"
"precise_timestamp": 1451714399123
Log.Time = 1451714399

Common time expressions

Note

Example	Time expression
2016/01/02 12:59:59	%Y/%m/%d %H:%M:%S
2016/01/02 12:59:59.1	%Y/%m/%d %H:%M:%S.%f
2016/01/02 12:59:59.987654321 +0700 (UTC)	%Y/%m/%d %H:%M:%S.%f %z (%Z)
2016/Jan/02 12:59:59,123456	%Y/%b/%d %H:%M:%S,%f
2019-07-15T04:16:47:123Z	%Y-%m-%dT%H:%M:%S:%f