The Nginx log format and directory are generally in the configuration file
The log configuration file defines the print format of Nginx logs, that is, the main format:
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$request_time $request_length '
'$status $body_bytes_sent "$http_referer" '
The declaration uses the main log format and the written file name.
access_log /var/logs/nginx/access.log main
Field name Description remoteaddr The IP address of the client. remote_user The username of the client. request The requested URL and HTTP protocol. status The request status. bodybytessent The number of bytes (not including the size of the response header) sent to the client. The total number of bytes for this variable is the same as that sent to the client by bytes_sent in modlogconfig of the Apache module. connection The connection serial number. connection_requests The number of requests received by using a connection. msec The log write time, which is measured in seconds and precise to milliseconds. pipe Whether or not requests are sent by using the HTTP pipeline.
pindicates requests are sent by using the HTTP pipeline. Otherwise, the value is
httpreferer The Web page link from which the access is directed. “http_user_agent” Information about the browser on the client. http_user_agent must be enclosed in double quotation marks. requestlength The length of a request, including the request line, request header, and request body. request_time The request processing time, which is measured in seconds and precise to milliseconds. The time starts when the first byte is sent to the client and ends when the logs are written after the last character is sent to the client. [$time_local] The local time in the general log format. This variable must be enclosed in brackets.
192.168.1.2 - - [10/Jul/2015:15:51:09 +0800] "GET /ubuntu.iso HTTP/1.0" 0.000 129 404 168 "-" "Wget/1.11.4 Red Hat modified"
For the complete process of collecting Nginx logs by using Logtail, see Quick start. Select the corresponding configuration based on your network deployment and actual situation. This document only shows how to configure data source in Step 3 Configure data import wizard in details.
Enter the Configuration Name and Log Path. Then, select Full Mode as the log collection mode.
Enter the log sample and turn on the Extract Field switch.
Click Manually Input Regular Expression to adjust the automatically generated regular expression.
Log Service supports automatically parsing the log sample with mouse-selection, that is, automatically generating the regular expression for the field selected with the mouse. However, considering the format of the actual log data might have minor changes, you must adjust the automatically generated regular expression according to the actual situations to conform to all the log formats that might occur in the collection process.
body_bytes_sentfields here are numeric, but can be a hyphen (-) in some situations. Therefore, replace the matched result
(\S+). If other fields are in the same situation, replace the matched result with the same rule.
Regular expression that conforms to the log format:
Click Validate after modifying the regular expression. If the regular expression is correct, extracted results are displayed. Adjust the regular expression if any errors exist.
Enter the corresponding key for the log content extraction result.
Assign a meaningful field name for each extraction result, for example, name a time field as time.
Turn on the Use System Time switch and then click Next.
After configuring Logtail, apply the configuration to the machine group to start collecting Nginx logs standardly.