Logtail plug-in that is used to parse Apache logs - - Alibaba Cloud Documentation Center

You can use a Logtail plug-in to parse Apache logs into structured data based on the log format that you specify in your Apache configuration file. In this case, a log is parsed into multiple key-value pairs.

Introduction to Apache logs

Apache servers generate Apache logs based on the log display format, log file directory, and log file name that are specified in an Apache configuration file. For example, if you enter CustomLog "/var/log/apache2/access_log" combined, logs are displayed in the combined log format, and the log file directory is /var/log/apache2/access_log.

Apache log formats

Combined log format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Common log format:
```
LogFormat "%h %l %u %t \"%r\" %>s %b" 
```

Custom log format:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized

The following table describes the log fields. For more information, see mod_log_config.

Format	Field	Description
%a	client_addr	The IP address of the client.
%A	local_addr	The local IP address.
%b	response_size_bytes	The number of bytes in a response. If no bytes are sent, a hyphen (-) is displayed for this field.
%B	response_bytes	The number of bytes in a response. If no bytes are sent, the digit 0 is displayed for this field.
%D	request_time_msec	The time that is required to process a request. Unit: microseconds.
%f	filename	The file name.
%h	remote_addr	The name of the remote host.
%H	request_protocol_supple	The request protocol.
%I	bytes_received	The number of bytes that are received by the server. This field is recorded in logs only after you enable the mod_logio module.
%k	keep_alive	The number of keep-alive requests that are handled on the connection.
%l	remote_ident	The information that is provided by the remote host for identification.
%m	request_method_supple	The HTTP request method.
%O	bytes_sent	The number of bytes that are sent by the server. This field is recorded in logs only after you enable the mod_logio module.
%p	remote_port	The port number of the server.
%P	child_process	The ID of the child process.
%q	request_query	The query string. If no query strings exist, an empty string is displayed.
%r	request	The first line of the request. This line consists of the HTTP request method, address, and HTTP version.
%R	response_handler	The type of the handler that generates a response on the server.
%s	status	The initial HTTP status of a response.
%>s	status	The final HTTP status of a response.
%t	time_local	The point in time at which the server receives the request.
%T	request_time_sec	The time that is required to process a request. Unit: seconds.
%u	remote_user	The username that is used by the client to send a request.
%U	request_uri_supple	The URI in a request. The URI does not include the query string.
%v	server_name	The name of the server.
%V	server_name_canonical	The name of the server. The name is specified by using the UseCanonicalName directive.
“%{User-Agent}i”	http_user_agent	The information about the client.
“%{Referer}i”	http_referer	The URL of the source web page.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview.

Configuration description

Parameter	Description
Log Format	The log format that is specified in the Apache configuration file. Valid values: common, combined, and Custom.
APACHE Logformat Configuration	The log configuration section that is specified in the Apache configuration file. In most cases, the section starts with LogFormat. If you set Log Format to common or combined, the system automatically inserts a value into this parameter. Check whether the value is the same as that specified in the Apache configuration file. If you set Log Format to Custom, specify a value based on your business requirements. For example, you can enter `LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized`.
Original Field	The original field that stores log content before parsing. Default value: content.
Regular Expression	The regular expression that is used to extract Apache logs. Simple Log Service automatically generates a regular expression based on APACHE Logformat Configuration.
Extracted Field	The key that is automatically extracted based on APACHE Logformat Configuration.
Retain Original Field if Parsing Fails	After you select Retain Original Field if Parsing Fails, the original field is retained if parsing fails.
Retain Original Field if Parsing Succeeds	After you select Retain Original Field if Parsing Succeeds, the original field is retained if parsing is successful.
New Name of Original Field	If you select Retain Original Field if Parsing Fails or Retain Original Field if Parsing Succeeds, you can rename the original field that stores log content.