Analyze IIS access logs

IIS is an extensible web server used to build and host websites. You can use access logs collected by IIS to obtain data such as page views, unique visitors, client IP addresses, bad requests, and network flow, to monitor and analyze access to your website.

logExtFileFlags="Date, Time, ClientIP, UserName, SiteName, ComputerName, ServerIP, Method, UriStem, UriQuery, HttpStatus, Win32Status, BytesSent, BytesRecv, TimeTaken, ServerPort, UserAgent, Cookie, Referer, ProtocolVersion, Host, HttpSubStatus"

1. Start the Data Import Wizard steps.
   1. On the homepage of the Log Service console, click the specified Project Name to enter the Logstore List page.
   2. Click the icon in the Data Import Wizard column of the specified project.
2. In Step 1 Select Data Source, choose IIS ASSESSLOG under the Third-Party Software category.
3. Configure the data source.
   1. Enter the Configuration Name and Log Path.
      You can view the log path in the IIS Manager.
4. Select Log format.
   Select the log format of your IIS access log.

   • IIS: Microsoft IIS log file format
   • NCSA: NCSA Common log file format
   • W3C: W3C Extended log file format
5. Fill in the field IIS Logformat configuration.
   • Microsoft IIS and NCSA Public formats have fixed configurations.
   • To configure the IIIS access log to W3C format, follow these steps:
   1. Open the IIS configuration file.
      • The default path for IIS5 configuration file: C:\WINNT\system32\inetsrv\MetaBase.bin
      • The default path for IIS6 configuration file: C:\WINDOWS\system32\inetsrv\MetaBase.xml
      • The default path for IIS7 configuration file: C:\Windows\System32\inetsrv\config\applicationHost.config

      
      
      

   2. As shown in figure 3, copy the text inside the quotation marks in the field logFile logExtFileFlags.
   3. In the field IIS Logformat configuration in the console, paste the specified text inside the quotation marks.

      
      
      

6. Confirm the key names.
   IIS log service will automatically extract the key names.

   
   
   

7. Advanced Options (Optional)

   Parameter	Description
   Upload Raw Log	Specifies whether to upload the raw log. If you turn on this switch, the raw log content is uploaded as the __raw__ field with the parsed log content.
   Topic Generation Mode	Null - Do not generate topic: The default value, which specifies that the topic is set to a null string. You can query logs without entering the topic. Machine Group Topic Attributes: sets the topic based on a machine group to differentiate log data generated on different frontend servers. File Path RegEx: uses Custom RegEx to extract a part of the log path as the topic. This mode is used to differentiate log data generated by different users or instances.
   Custom RegEx	The custom regular expression specified if you set Topic Generation Mode to File Path RegEx.
   Log File Encoding	utf8: specifies UTF-8 encoding. gbk: specifies GBK encoding.
   Maximum Directory Monitoring Depth	The maximum depth of the monitored directory when logs are collected from the log source, that is, at most how many levels of directories can be monitored. Valid values: [0, 1000]. A value of 0 indicates that only the current directory is monitored.
   Timeout	Specifies whether the system considers that a log file has timed out if the file is not updated within the specified period. You can set Timeout as follows: Never: specifies that all log files are continuously monitored without timeout. 30 Minute Timeout: specifies that if a log file is not updated within 30 minutes, the system considers that the log file has timed out and no longer monitors the file.
   Filter Configuration	The filter conditions that logs must completely meet before they can be collected. For example: Collect logs that meet a condition: Set a condition Key:level Regex:WARNING|ERROR, which indicates that only logs whose level is WARNING or ERROR are collected. Filter logs that do not meet a condition: Set a condition Key:level Regex:^(?!.*(INFO|DEBUG)).* , which indicates that logs whose level is INFO or DEBUG are not collected. Set a condition Key:url Regex:.*^(?!.*(healthcheck)).* , which indicates that logs with healthcheck in url are not collected. For example, logs in which the key is url and the value is /inner/healthcheck/jiankong.html are not collected. For more examples, see regex-exclude-word and regex-exclude-pattern.

   Confirm configurations and click Next.
8. Apply the configuration to the machine group.
   Select the machine groups to which the specified configurations will apply. Click Apply to Machine Group at the bottom right corner of the page.

   If you have not created any machine group, click Create Machine Group to create one.

9. Configure Search, Analysis, and Visualization (Optional).
   When the heartbeat status of the machine group is normal, you can click Preview to view log data.

   
   
   

   Confirm your Index Properties in the current page to view and analyze the collected log data. Click Open to view the Key/Value Index Attributes.

   You can configure key name mapping. The key names are generated based on previewed data, and correspond to the default key names.

   
   
   

   The system provides the default dashboard LogstoreName-iis-dashboard for you. After the preceding configurations are completed, you can view real-time data (including client IP distribution and the proportion of each HTTP status) on the dashboard.

   • Use the following statement to obtain client IP distribution:

     | select ip_to_geo("c-ip") as country, count(1) as c group by ip_to_geo("c-ip") limit 100

   • Use the following statement to check recent page views and unique visitors:

     *| select approx_distinct("c-ip") as uv ,count(1) as pv , date_format(date_trunc('hour', __time__), '%m-%d %H:%i') as time group by date_format(date_trunc('hour', __time__), '%m-%d %H:%i') order by time limit 1000

     
     
     

   • Use the following statement to obtain the proportion of each HTTP status:

     *| select count(1) as pv ,"sc-status" group by "sc-status"

     
     
     

   • Use the following statement to view the network flow:

     *| select sum("sc-bytes") as net_out, sum("cs-bytes") as net_in ,date_format(date_trunc('hour', time), '%m-%d %H:%i') as time group by date_format(date_trunc('hour', time), '%m-%d %H:%i') order by time limit 10000

     
     
     

   • Use the following statement to get the proportion of each Hrequest methodTTP request method:

     *| select count(1) as pv ,"cs-method" group by "cs-method"

     
     
     

   • Use the following statement to obtain the proportion of each browser type:

     *| select count(1) as pv, case when "user-agent" like '%Chrome%' then 'Chrome' when "user-agent" like '%Firefox%' then 'Firefox' when "user-agent" like '%Safari%' then 'Safari' else 'unKnown' end as "user-agent" group by case when "user-agent" like '%Chrome%' then 'Chrome' when "user-agent" like '%Firefox%' then 'Firefox' when "user-agent" like '%Safari%' then 'Safari' else 'unKnown' end order by pv desc limit 10

     
     
     

   • Use the following statement to view the top 10 most visited site addresses:

     *| select count(1) as pv, split_part("cs-uri-stem",'?',1) as path group by split_part("cs-uri-stem",'?',1) order by pv desc limit 10