Analyze IIS access logs

Internet Information Services (IIS) is an extensible web server used to build and host websites. You can collect and analyze IIS access logs to obtain data such as page views (PVs), unique visitors (UVs), client IP distribution, error requests, and inbound and outbound traffic, to monitor and analyze access to your website.

logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus"

1. Start the Import Data wizard.
   1. Log on to the Log Service console. On the homepage, click the target project name.
   2. In the left-side navigation pane, find the target Logstore and click the plus sign (+) next to Data Import.
2. In the Import Data dialog box, select IIS - Text Log.
3. Select a Logstore.
   You can select an existing Logstore, or create a project and a Logstore.
4. Create a server group.
   Before creating a server group, make sure that Logtail is installed.

   • Servers of Alibaba Group: By default, Logtail is installed on these servers. If Logtail is not installed on a server, contact Alibaba Cloud as prompted.
   • ECS instances: Select ECS instances and click Install. ECS instances that are running in Windows do not support one-click installation of Logtail. In this case, you need to manually install Logtail. For more information, see Install Logtail in Windows.
   • On-premises servers: Install Logtail as prompted. For more information about how to install Logtail, see Install Logtail in Linux or Install Logtail in Windows based on your operating system.

   After installing Logtail, click Complete Installation to create a server group. If you have created a server group, click Use Existing Server Groups.
5. Configure the server group.
   Select a server group and move the server group from Source Server Groups to Applied Server Groups.
6. Create a Logtail configuration.
   1. Set Config Name and Log Path.
      You can view the log path in IIS Manager.
   2. Set Log format.
      Select the log format of your IIS access logs. Valid values:

      • IIS: Microsoft IIS log file format
      • NCSA: NCSA common log file format
      • W3C: W3C extended log file format
   3. Set IIS Configuration.
      • If you select the Microsoft IIS log file format or NCSA common log file format, the IIS configuration is preset.
      • If you select the W3C extended log file format, follow these steps to set IIS Configuration:
        1. Open the IIS configuration file.
           • Default path of the IIS5 configuration file: C:\WINNT\system32\inetsrv\MetaBase.bin
           • Default path of the IIS6 configuration file: C:\WINDOWS\system32\inetsrv\MetaBase.xml
           • Default path of the IIS7 configuration file: C:\Windows\System32\inetsrv\config\applicationHost.config

           

        2. In the IIS configuration file, copy the text inside the quotation marks (" ") in the logFile logExtFileFlags field.
        3. Paste the text to the quotation marks (" ") in the IIS Configuration field.

           

   4. Confirm IIS key names.
      Log Service automatically extracts the key names from the IIS log format.

      

   5. Set Drop Failed to Parse Logs.
      Specify whether to upload logs that fail to be parsed to Log Service.

      If you turn on this switch, the logs that fail to be parsed are not uploaded to Log Service. If you turn off this switch, the raw logs are uploaded to Log Service when log parsing fails. The key of a raw log is __raw_log__ and the value is the log content.

   6. Optional. Set advanced options as required.

      Parameter	Description
      Upload Raw Log	Specifies whether to upload the raw log. If you turn on this switch, the raw log content is uploaded as the __raw__ field with the parsed log content.
      Topic Generation Mode	Null - Do not generate topic: The default value, which specifies that the topic is set to a null string. You can query logs without entering the topic. Machine Group Topic Attributes: sets the topic based on a machine group to differentiate log data generated on different frontend servers. File Path RegEx: uses Custom RegEx to extract a part of the log path as the topic. This mode is used to differentiate log data generated by different users or instances.
      Custom RegEx	The custom regular expression specified if you set Topic Generation Mode to File Path RegEx.
      Log File Encoding	utf8: specifies UTF-8 encoding. gbk: specifies GBK encoding.
      Maximum Directory Monitoring Depth	The maximum depth of the monitored directory when logs are collected from the log source, that is, at most how many levels of directories can be monitored. Valid values: [0, 1000]. A value of 0 indicates that only the current directory is monitored.
      Timeout	Specifies whether the system considers that a log file has timed out if the file is not updated within the specified period. You can set Timeout as follows: Never: specifies that all log files are continuously monitored without timeout. 30 Minute Timeout: specifies that if a log file is not updated within 30 minutes, the system considers that the log file has timed out and no longer monitors the file.
      Filter Configuration	The filter conditions that logs must completely meet before they can be collected. For example: Collect logs that meet a condition: Set a condition Key:level Regex:WARNING|ERROR, which indicates that only logs whose level is WARNING or ERROR are collected. Filter logs that do not meet a condition: Set a condition Key:level Regex:^(?!.*(INFO|DEBUG)).* , which indicates that logs whose level is INFO or DEBUG are not collected. Set a condition Key:url Regex:.*^(?!.*(healthcheck)).* , which indicates that logs with healthcheck in url are not collected. For example, logs in which the key is url and the value is /inner/healthcheck/jiankong.html are not collected. For more examples, see regex-exclude-word and regex-exclude-pattern.

      Confirm the configuration and click Next.
7. Configure log query and analysis.
   By default, Log Service creates key names for you. You can set the actual key names based on previewed data and map them to the default key names. To create or modify indexes, choose Index Attributes > Modify on the Search & Analysis page.

   You can preview the collected log data if servers in the serve group have normal heartbeats.

   

   Log Service creates a default dashboard named LogstoreName-iis-dashboard for you. After the preceding configuration is completed, you can view real-time data on the dashboard, including the distribution of client IP addresses and the percentage of each HTTP status code.

   • Client IP distribution: Use the following SQL statement to collect statistics on the distribution of client IP addresses:

     | select ip_to_geo("c-ip") as country, count(1) as c group by ip_to_geo("c-ip") limit 100

   • PVs and UVs: Use the following SQL statement to count the number of PVs and UVs:

     *| select approx_distinct("c-ip") as uv ,count(1) as pv , date_format(date_trunc('hour', __time__), '%m-%d %H:%i') as time group by date_format(date_trunc('hour', __time__), '%m-%d %H:%i') order by time limit 1000

     

   • Percentages of HTTP status codes: Use the following SQL statement to count the percentage of each HTTP status code returned:

     *| select count(1) as pv ,"sc-status" group by "sc-status"

     

   • Inbound and outbound traffic: Use the following SQL statement to collect statistics on the inbound and outbound traffic:

     *| select sum("sc-bytes") as net_out, sum("cs-bytes") as net_in ,date_format(date_trunc('hour', time), '%m-%d %H:%i') as time group by date_format(date_trunc('hour', time), '%m-%d %H:%i') order by time limit 10000

     

   • Percentages of request methods: Use the following SQL statement to count the percentage of each request method used:

     *| select count(1) as pv ,"cs-method" group by "cs-method"

     

   • Percentages of browser types: Use the following SQL statement to count the percentage of each browser type used:

     *| select count(1) as pv, case when "user-agent" like '%Chrome%' then 'Chrome' when "user-agent" like '%Firefox%' then 'Firefox' when "user-agent" like '%Safari%' then 'Safari' else 'unKnown' end as "user-agent" group by case when "user-agent" like '%Chrome%' then 'Chrome' when "user-agent" like '%Firefox%' then 'Firefox' when "user-agent" like '%Safari%' then 'Safari' else 'unKnown' end order by pv desc limit 10

     

   • Top 10 most visited pages: Use the following SQL statement to count the top 10 visited pages with the most PVs:

     *| select count(1) as pv, split_part("cs-uri-stem",'?',1) as path group by split_part("cs-uri-stem",'?',1) order by pv desc limit 10