The Nginx log format and directory are specified in the
/etc/nginx/nginx.conf configuration file.
The log configuration file defines the print formats as main format:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$request_time $request_length '
'$status $body_bytes_sent "$http_referer" '
The following statement defines how to use the main log format and defines the name of the file where to write the log data.
access_log /var/logs/nginx/access.log main
Field Description remoteaddr The IP address of the agent. remote_user The username of the agent. request The requested URL and HTTP protocol. status The request status. bodybytessent The number of bytes (not including the size of the response header) sent to the agent. This variable is used with bytes_sent in modlogconfig of the Apache module. connection The connection serial number. connection_requests The number of requests received over a connection. msec The log write time, measured in seconds and precise to milliseconds. pipe Whether requests are sent through the HTTP pipeline.
pindicates requests are sent through the HTTP pipeline.
.indicates requests are not sent through the HTTP pipeline.
httpreferer The webpage link from which access is directed. “http_user_agent” Information about the agent’s browser. http_user_agent must be enclosed by double quotation marks. requestlength The length of a request, including the request line, request header, and request body. request_time The request processing time, which is measured in seconds and precise to milliseconds. The time starts when the first byte is sent to the agent and ends when the logs are written after the last character is sent to the agent. [$time_local] The local time when the general log format is applied. This variable must be enclosed in brackets.
192.168.1.2 - - [10/Jul/2015:15:51:09 +0800] "GET /ubuntu.iso HTTP/1.0" 0.000 129 404 168 "-" "Wget/1.11.4 Red Hat modified"
For details about the standard process, refer to Quick start. Choose the corresponding configuration based on your network deployment and actual situation. This doucumentaion only shows detailed complete configuration of second step Specify Collection Mode.
Enter the configuration name, log path, and choose Full Mode for Specify Collection Mode.
Enter the sample log, and open Extract Field.
Click Manually Input Regular Expression, and manually edit the regular expression after it is automatically generated.
Log Service support automatically generation of regular expression, that is, generate regular expression by mouse-selection. But the automatically generated results are for reference only. You need to manually edit the generated regular expression to fit all the logs that might be selected by Logtail.
lengthfield is numeric, but can be filled in with a hyphen. In this case, replace the matched result (
\d+) with (
\S+). If other fields do not belong to the defined type, make similar replacements.
Regular Expression for this case:
After replacing, click Validate. If the regular expression is correct, extracted results are displayed. Manually adjust the regular expression if it is incorrect.
Enter the corresponding Key for the log content extraction result.
Assign a defined field to each extracted result. For example, name a field “time”. Click Auto Generate next to Time Format. Then click Next.
After Logtail configuration is completed, push the configuration to the client.