This topic describes how to configure the Apache mode in the Log Service console to collect logs.

Log formats

Apache logs can be written in the combined or common format. You can also customize the log format.
  • Syntax of the combined log format: 
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
  • Syntax of the common log format: 
    LogFormat "%h %l %u %t \"%r\" %>s %b"
  • Syntax of a custom log format: 
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %f %k %p %q %R %T %I %O" customized
Format field Key name Description
%a client_addr The IP address of the client.
%A local_addr The local IP address.
%b response_size_bytes The size of the response. Unit: bytes. The response format is CLF. If no response is returned, the value of this parameter is -.
%B response_bytes The size of the response. Unit: bytes. If no response is returned, the value of this parameter is 0.
%D request_time_msec The period of time for which the request is processed. Unit: milliseconds.
%h remote_addr The name of the remote host.
%H request_protocol_supple The protocol of the request.
%l remote_ident The identity information that is provided by a remote host.
%m request_method_supple The method of the request.
%p remote_port The port number of the server.
%P child_process The ID of the child process.
%q request_query The string that is used to query logs. If no query string exists, the value is an empty string.
“%r” request The content of the request. The content includes the method name, address, and HTTP protocol fields of the request.
%s status The HTTP status code of the response.
%>s status The HTTP status code of the final response.
%f filename The name of the requested file.
%k keep_alive The number of keep-alive requests.
%R response_handler The type of the handler that generates the response on the server.
%t time_local The local time when the server receives the request.
%T request_time_sec The time period for which the request is processed. Unit: seconds.
%u remote_user The username that you used to log on to the client.
%U request_uri_supple The URI of the request. The URI does not include the string that is used to query logs.
%v server_name The name of the server.
%V server_name_canonical The name of the server. The name is specified by using the UseCanonicalName directive.
%I bytes_received The number of bytes that are received by the server. To use this key, you must enable the mod_logio module.
%O bytes_sent The number of bytes that are sent by the server. To use this key, you must enable the mod_logio module.
“%{User-Agent}i” http_user_agent The information of the client.
“%{Rererer}i” http_referer The URL of the web page. The URL is linked to the resource that is being requested.
During Logtail configuration for Apache logs, you must specify the log format, log file path, and log file name. For example, the following command indicates that the logs are written in the combined format. The path of the log file is /var/log/apache2/access_log. 
CustomLog "/var/log/apache2/access_log" combined
An Apache log entry is as follows: 
192.168.1.2 - - [02/Feb/2016:17:44:13 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "http://localhost/x1.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36"

Procedure

  1. Log on to the Log Service console.
  2. In the Import Data section, click Apache - Text Log.
  3. In the Specify Logstore step, select the target project and Logstore, and click Next.
    You can also click Create Now to create a project and a Logstore. For more information, see Step 1: Create a project and a Logstore.
  4. In the Create Machine Group step, create a machine group.
    • If a machine group is available, click Using Existing Machine Groups.
    • If no machine group is available, perform the following steps. The following steps take ECS instances as an example to describe how to create a machine group:
      1. Click the ECS Instances tab. On the tab, select the ECS instances that you want to add to a machine group and then click Install.

        If Logtail is installed on the ECS instances, click Complete Installation.

        Note
        • If the ECS instances run Linux, click Install and Logtail is automatically installed on the ECS instances.
        • If the ECS instances run Windows, you must manually install Logtail on the ECS instances. For more information, see Install Logtail in Windows.
        • If you want to collect logs from a user-created cluster, you must manually install Logtail on the servers in the cluster. For more information, see Install Logtail in Linux or Install Logtail in Windows.
      2. After the installation is completed, click Complete Installation.
      3. On the page that appears, set relevant parameters for the machine group. For more information, see Create an IP address-based machine group or Create a custom ID-based machine group.
  5. In the Machine Group Settings step, apply Logtail configurations to the machine group.
    Select the created machine group and move the group from Source Machine Groups to Applied Machine Groups.
  6. In the Logtail Config step, create a Logtail configuration.
    Parameter Description
    Config Name The name of the Logtail configuration. The name cannot be modified after the Logtail configuration is created.

    You can also click Import Other Configuration to import Logtail configurations from other projects.

    Log Path The directory and name of the log file.
    The specified log file names can be complete file names or file names that contain wildcards. For more information about wildcards that can be used in patterns of directory and file names, visit Wildcard matching. The log files in all levels of subdirectories under the specified directory that match the specified pattern are monitored. Examples:
    • /apsara/nuwa/ ... /*.log indicates the files whose extension is .log in the /apsara/nuwa directory and its subdirectories are monitored.
    • /var/logs/app_* ... /*.log* indicates each file that meets the following conditions is monitored: The file name contains .log. The file is stored in a subdirectory (at any level) of the /var/logs directory. The name of the subdirectory matches the app_* pattern.
    Note
    • A log file can be collected by using only one Logtail configuration file.
    • You can include only asterisks (*) and question marks (?) as wildcard characters in the log path.
    Blacklist If this switch is turned on, you can configure a blacklist in the Add Blacklist section. You can configure a blacklist to skip the specified directories or files during log collection. The names of the specified directories and files support exact match and wildcard match. Examples:
    • If you select Filter by Directory from the Filter Type drop-down list and enter /tmp/mydir in the Content column, all files in the directory are skipped.
    • If you select Filter by File from the Filter Type drop-down list and enter /tmp/mydir/file in the Content column, only the specified file in the directory are skipped.
    Docker File If the file in the Docker container is a log file, you can directly specify the log path and container tags. Logtail automatically monitors the creation and destruction of containers, and collects log entries of the specified containers based on the specified tags. For more information about container text logs, see Use the console to collect Kubernetes text logs in the DaemonSet mode.
    Mode The default value is Apache Configuration Mode. You can select other modes. For more information about how to configure other modes, see Overview.
    Log format Select a log format based on the format that are configured in your Apache log configuration file. To facilitate the query and analysis of log data, we recommend that you select Custom.
    APACHE Logformat Configuration Enter the log configuration section that is specified in the Apache configuration file. The section starts with the LogFormat field.

    If you set Log Format to common or combined, the syntax of the corresponding log format is automatically generated. You must check whether the format of the generated logs is the same as the log format that is specified in the Apache configuration file.

    APACHE Key Name Log Service reads the Apache key names from the content in the APACHE Logformat Configuration section.
    Drop Failed to Parse Logs
    • If you turn on the Drop Failed to Parse Logs switch, logs that fail to be parsed cannot be uploaded to Log Service.
    • If you turn off the Drop Failed to Parse Logs switch, raw logs are uploaded when the raw logs fail to be parsed.
    Maximum Directory Monitoring Depth The maximum number of directory layers that can be recursively monitored during log collection. Valid values: 0 to 1000. The value 0 indicates that only the directory specified in the log path is monitored.
    Specify Advanced Options based on your business requirements. We recommend that you do not modify the default settings unless otherwise required.
    Parameter Description
    Enable Plug-in Processing Specifies whether to use plug-ins for Logtail to process logs. If you turn on the switch, plug-ins are used to process logs.
    Upload Raw Log If you turn on this switch, raw logs are written to the __raw__ field and uploaded with the parsed logs.
    Topic Generation Mode
    • Null - Do not generate topic: This mode is selected by default. In this mode, the topic field is set to an empty string and you can query logs without the need to enter a topic.
    • Machine Group Topic Attributes: This mode is used to differentiate log data that is generated by different servers.
    • File Path Regex: If you select File Path Regex for Topic Generation Mode, you must configure a regular expression in the Custom RegEx field. The part of a log path that matches the regular expression is used as the topic name. This mode is used to differentiate log data that is generated by different users or ECS instances.
    Log File Encoding
    • utf8: indicates UTF-8 encoding.
    • gbk: indicates GBK encoding.
    Timezone The time zone where logs are collected.
    • System Timezone: This option is selected by default. It indicates that the time zone where logs are collected is the same as the time zone to which the server belongs.
    • Custom: Select a time zone.
    Timeout If a log file is not updated within the specified period of time, Logtail considers the file to be timed out.
    • Never: All log files are continuously monitored and never time out.
    • 30 Minute Timeout: If a log file is not updated within 30 minutes, Logtail considers the log file to be timed out and no longer monitors the file.

      If you select 30 Minute Timeout, you must set Maximum Timeout Directory Depth. Valid values: 1 to 3.

    Filter Configuration Specifies to collect logs that match the filtering conditions. Examples:
    • If you want to collect only the logs with the severity level of WARNING or ERROR, set the condition Key:level Regex:WARNING|ERROR. It indicates that logs with the severity level of WARNING or ERROR are collected.
    • Filter logs that do not meet a condition:
      • Set the condition to Key:level Regex:^(?!. *(INFO|DEBUG)). * if you want to filter out the logs with the severity level of INFO or DEBUG.
      • Set the condition to Key:url Regex:. *^(?!.*(healthcheck)). * if you want to filter out the logs whose URL contains the keyword healthcheck. For example, logs in which the value of the url key is /inner/healthcheck/jiankong.html are not collected.

    For more examples, see regex-exclude-word and regex-exclude-pattern.

  7. In the Configure Query and Analysis step, set indexes.
    Indexes are configured by default. You can reconfigure the indexes as needed. For more information, see Enable and configure the index feature for a Logstore.
    Note
    • You must set Full Text Index or Field Search. If you set both of them, the settings of Field Search prevail.
    • If the data type of an index is Long or Double, the Case Sensitive and Delimiter settings are unavailable.

After you complete the settings, Log Service starts to collect logs.