Collect logs in Apache mode

Procedure

1. Log on to the Log Service console.
2. In the Import Data section, click Apache - Text Log.
3. Select the project and Logstore. Then, click Next.
4. Create a machine group.
   • If a machine group is available, click Using Existing Machine Groups.
   • If no machine groups are available, perform the following steps to create a machine group. In this example, an Elastic Compute Service (ECS) instance is used.
     1. On the ECS Instances tab, select Manually Select Instances. Then, select the ECS instance that you want to use and click Execute Now.

        For more information, see Install Logtail on ECS instances.

        Note If you want to collect logs from self-managed clusters or servers from third-party cloud service providers, you must manually install Logtail. For more information, see Install Logtail on a Linux server or Install Logtail in Windows.
     2. After Logtail is installed, click Complete Installation.
     3. In the Create Machine Group step, configure Name and click Next.

        Log Service allows you to create IP address-based machine groups and custom identifier-based machine groups. For more information, see Create an IP address-based machine group and Create a custom ID-based machine group.

5. Select the newly created machine group and move it from the Source Server Groups section to the Applied Server Groups section. Then, click Next.
   Notice If you apply a machine group immediately after it is created, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. In this case, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
6. In the Logtail Config step, create a Logtail configuration.
   

   Parameter	Description
   Config Name	Enter a name for the Logtail configuration. The name must be unique in a project. After the Logtail configuration is created, you cannot change the name of the Logtail configuration. You can click Import Other Configuration to import a Logtail configuration from another project.
   Log Path	Specify the log file directory and log file name. You can specify an exact directory and an exact name. You can also use wildcards to specify the directory and name. For more information, see Wildcard matching. Log Service scans all levels of the specified directory to match log files. Examples: If you specify /apsara/nuwa/**/*.log, Log Service matches the files whose name is suffixed by .log in the /apsara/nuwa directory and its recursive subdirectories. If you specify /var/logs/app_*/*.log, Log Service matches the files that meet the following conditions: The file name contains .log. The file is stored in a subdirectory under /var/logs or in a recursive subdirectory of the subdirectory. The name of the subdirectory matches the app_* pattern. Note By default, logs in each log file can be collected by using one Logtail configuration. If you want to collect logs in a log file by using multiple Logtail configurations, you must create a symbolic link for the directory in which the log file is stored. For example, if you want to collect logs from the /home/log/nginx/log/log.log file by using two Logtail configurations, you must run the following command to create a symbolic link that points to the directory of the file. Then, specify the real path in one Logtail configuration and specify the symbolic link in the other Logtail configuration. ln -s /home/log/nginx/log /home/log/nginx/link_log When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcards.
   Blacklist	If you turn on Blacklist, you must configure a blacklist to specify the directories or files that you want Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcards to specify directories and file names. Examples: If you select Filter by Directory from a drop-down list in the Filter Type column and enter /home/admin/dir1 for Content, all files in the /home/admin/dir1 directory are skipped. If you select Filter by Directory from a drop-down list in the Filter Type column and enter /home/admin/dir* for Content, the files in all subdirectories whose names are prefixed by dir in the /home/admin/ directory are skipped. If you select Filter by Directory from a drop-down list in the Filter Type column and enter /home/admin/*/dir for Content, all files in dir directories in each subdirectory of the /home/admin/ directory are skipped. For example, the files in the /home/admin/a/dir directory are skipped, but the files in the /home/admin/a/b/dir directory are not skipped. If you select Filter by File from a drop-down list in the Filter Type column and enter /home/admin/private*.log for Content, all files whose names are prefixed by private and suffixed by .log in the /home/admin/ directory are skipped. If you select Filter by File from a drop-down list in the Filter Type column and enter /home/admin/private*/*_inner.log for Content, all files whose names are suffixed by _inner.log in the subdirectories whose names are prefixed by private in the /home/admin/ directory are skipped. For example, the /home/admin/private/app_inner.log file is skipped, but the /home/admin/private/app.log file is not skipped. Note When you configure the blacklist, you can use only asterisks (*) or question marks (?) as wildcards. If you use wildcards to configure Log Path and want to skip some directories in the specified directory, you must configure the blacklist and enter a complete directory. For example, if you set Log Path to /home/admin/app*/log/*.log and want to skip all subdirectories in the /home/admin/app1* directory, you must configure the blacklist by selecting Filter by Directory and entering /home/admin/app1*/**. If you enter /home/admin/app1*, the blacklist cannot take effect. Computational overhead is generated when the blacklist is used. We recommend that you add a maximum of 10 entries to the blacklist.
   Docker File	If you want to collect logs from Docker containers, you must turn on Docker File and specify the directories and tags of the containers. Logtail monitors the containers to check whether the containers are created or destroyed. Then, Logtail filters the logs of the containers based on tags and collects the logs that meet the filter conditions. For more information about how to collect the text logs of containers, see Use the console to collect Kubernetes text logs in DaemonSet mode.
   Mode	Select the log collection mode. By default, Apache Configuration Mode is displayed. You can change the mode.
   Log format	Select the log format that is specified in the Apache configuration file. Valid values: common, combined, and Custom.
   APACHE Logformat Configuration	Enter the log configuration section that is specified in the Apache configuration file. In most cases, the section starts with LogFormat. For more information, see Additional information: Log formats and sample logs. If you set Log format to common or combined, the system automatically inserts a value into this field. Check whether the value is the same as that specified in the Apache configuration file. If you set Log format to Custom, specify a value based on your business requirements. For example, you can enter LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized.
   APACHE Key Name	Log Service automatically reads Apache keys from the value of the APACHE Logformat Configuration field.
   Drop Failed to Parse Logs	Specify whether to drop the logs that fail to be parsed. If you turn on Drop Failed to Parse Logs, the logs that fail to be parsed are not uploaded to Log Service. If you turn off Drop Failed to Parse Logs, raw logs are uploaded to Log Service if the logs fail to be parsed.
   Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. Valid values: 0 to 1000. The value 0 indicates that only the specified log file directory is monitored.

   You can configure advanced settings based on your business requirements. We recommend that you do not modify the advanced settings. The following table describes the parameters in the advanced settings.

   Parameter	Description
   Enable Plug-in Processing	If you turn on Enable Plug-in Processing, you can configure Logtail plug-ins to process logs. For more information, see Overview. Note If you turn on Enable Plug-in Processing, specific parameters such as Upload Raw Log, Timezone, Drop Failed to Parse Logs, Filter Configuration, and Incomplete Entry Upload (Delimiter mode) become unavailable.
   Upload Raw Log	If you turn on Upload Raw Log, each raw log is uploaded to Log Service as a value of the __raw__ field together with the log parsed from the raw log.
   Topic Generation Mode	Select the topic generation mode. For more information, see Log topics. Null - Do not generate topic: This is the default value. In this mode, the topic field is set to an empty string. When you query logs, you do not need to specify a topic. Machine Group Topic Attributes: In this mode, topics are configured at the machine group level. This mode is used to differentiate between the logs that are generated by different servers. File Path RegEx: In this mode, you must specify a regular expression in the Custom RegEx field. The part of a log path that matches the regular expression is used as the topic. This mode is used to differentiate between the logs that are generated by different users or instances.
   Log File Encoding	Select the encoding format of log files. Valid values: utf8 and gbk.
   Timezone	Select the time zone where logs are collected. Valid values: System Timezone: This is the default value. If you select this value, the time zone to which the server belongs is used. Custom: If you select this value, you must select a time zone.
   Timeout	Select a timeout period of log files. If a log file is not updated within the specified period, Logtail considers the file to be timed out. Valid values: Never: All log files are continuously monitored and never time out. 30 Minute Timeout: If a log file is not updated within 30 minutes, Logtail considers the file to be timed out and stops monitoring the file. If you select 30 Minute Timeout, you must configure the Maximum Timeout Directory Depth parameter. Valid values: 1 to 3.
   Filter Configuration	Specify the filter conditions that are used to collect logs. Only the logs that match the specified filter conditions are collected. Examples: Collect logs that meet specified conditions: If you set Key to level and RegEx to WARNING|ERROR, only the logs whose level is WARNING or ERROR are collected. Filter out logs that do not meet specified conditions. For more information, see Regular-Expressions.info. If you set Key to level and RegEx to ^(?!.*(INFO|DEBUG)).*, the logs whose level is INFO or DEBUG are not collected. If you set Key to level and RegEx to ^(?!(INFO|DEBUG)$).*, the logs whose level is INFO or DEBUG are not collected. If you set Key to url and RegEx to .*^(?!.*(healthcheck)).*, the logs whose URL contains healthcheck are not collected. For example, if a log has the Key field of url and the Value field of /inner/healthcheck/jiankong.html, the log is not collected For more information, see regex-exclude-word and regex-exclude-pattern.

   Click Next to complete the Logtail configuration. Then, Log Service starts to collect logs.

   Note

   • A Logtail configuration requires a maximum of 3 minutes to take effect.
   • If an error occurs when you use Logtail to collect logs, see Diagnose collection errors.
7. Preview collected logs, configure indexes, and then click Next.
   By default, Log Service enables full-text indexing. You can also configure field-based indexes on collected logs in manual or automatic mode. For more information, see Configure indexes.

   Note

   • If you want to query and analyze logs, you must enable full-text indexing or configure field-based indexes. If you configure both features, the system prioritizes the settings of field-based indexes.
   • If the data type of an index is long or double, you cannot configure the Case-Sensitive or Delimiter parameter.

Additional information: Log formats and sample logs

Before you collect Apache logs, you must specify the log display format, log file directory, and log file name. For example, if you enter CustomLog "/var/log/apache2/access_log" combined, logs are displayed in the combined log format, and the log file directory is /var/log/apache2/access_log. The following list describes supported log formats and provides a sample log:

• Log formats
  • The combined log format:

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

  • The common log format:

    LogFormat "%h %l %u %t \"%r\" %>s %b" 

  • A custom log format:

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized

  The following table describes the fields for these log formats. For more information, see mod_log_config.

  Field	Key	Description
  %a	client_addr	The IP address of the client.
  %A	local_addr	The local IP address.
  %b	response_size_bytes	The number of bytes in a response. If no bytes are sent, a hyphen (-) is displayed for this field.
  %B	response_bytes	The number of bytes in a response. If no bytes are sent, the digit 0 is displayed for this field.
  %D	request_time_msec	The time required to process a request. Unit: microseconds.
  %f	filename	The file name.
  %h	remote_addr	The name of the remote host.
  %H	request_protocol_supple	The request protocol.
  %I	bytes_received	The number of bytes that are received by the server. This field is recorded in logs only after you enable the mod_logio module.
  %k	keep_alive	The number of keep-alive requests handled on the connection.
  %l	remote_ident	The information that is provided by the remote host for identification.
  %m	request_method_supple	The HTTP request method.
  %O	bytes_sent	The number of bytes that are sent by the server. This field is recorded in logs only after you enable the mod_logio module.
  %p	remote_port	The port number of the server.
  %P	child_process	The ID of the child process.
  %q	request_query	The query string. If no query strings exist, an empty string is displayed.
  %r	request	The first line of a request. This line consists of the HTTP request method, address, and HTTP version.
  %R	response_handler	The type of the handler that generates a response on the server.
  %s	status	The initial HTTP status of a response.
  %>s	status	The final HTTP status of a response.
  %t	time_local	The point in time at which the server receives a request.
  %T	request_time_sec	The time required to process a request. Unit: seconds.
  %u	remote_user	The username that is used by the client to send a request.
  %U	request_uri_supple	The URI in a request. The URI does not include the query string.
  %v	server_name	The name of the server.
  %V	server_name_canonical	The name of the server. The name is specified by using the UseCanonicalName directive.
  "%{User-Agent}i"	http_user_agent	The information about the client.
  "%{Rererer}i"	http_referer	The URL of the source web page.

• Sample log

  192.168.1.2 - - [02/Feb/2020:17:44:13 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "http://localhost/x1.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36"