After you enable the log collection feature for the domain names that are added to WAF, you can use the log query feature to query and analyze the logs that are collected in real time. You can configure charts and create alert rules based on the query and analysis results.

Prerequisites

Query and analyze logs

  1. Log on to the Web Application Firewall console.
  2. In the top navigation bar, select the resource group and region to which the WAF instance belongs. The region can be Mainland China or International.
  3. In the left-side navigation pane, choose Log Management > Log Service.
  4. In the upper section of the Log Service page, select the domain name that you want to manage.
    Notice Make sure that log collection is enabled for the domain name. Otherwise, WAF does not collect the logs of the domain name, and you cannot query or analyze the logs of the domain name. To enable log collection, turn on Status.
    Domain name
  5. On the Log Query tab, execute a query statement to query and analyze WAF logs.
    1. Specify the query time range by using the time selector.
    2. Enter a query statement in the search box.
      Query statements use the syntax that is specific to Alibaba Cloud Log Service. For more information about the syntax, see Search syntax. The log fields that are included in WAF logs are used as query fields in the query statements. For more information about the log fields that are supported by WAF, see Log fields supported by WAF.
      If you do not know the query syntax, we recommend that you use Advanced Search. You need only to expand Advanced Search above the search box, specify search conditions, and click Search. The query statement is automatically generated based on the search conditions in the search box. Advanced SearchThe following table describes the search conditions that are supported by Advanced Search.
      Search condition Description
      IP The IP address of the client that sends the request.
      Trace ID The unique ID that is generated by WAF for each request. This ID is provided when WAF returns an error page or a response page that prompts the client to complete slider CAPTCHA verification to the client. You can use this ID to analyze and troubleshoot the error.
      Rule ID The ID of the WAF protection rule that is matched by the request. You can obtain the ID on the Security Report page or by choosing System Management > Protection Rule Group.
      Server Response Code The HTTP status code that is sent by the origin server as a response to the request forwarded by WAF.
      Status Code Returned by WAF The HTTP status code that is sent by WAF as a response to the request sent by the client.
      Protection Features The type of the WAF protection rule that is matched by the request. For more information about WAF protection rules and their configuration methods, see Overview.
    3. If you want to compute and analyze the query results, you must enter an analytic statement following the search statement in the search box. Otherwise, skip this step.
      Analytic statements and search statements are separated by vertical bars (|). The analytic statements use the standard SQL-92 syntax. For more information about the analytic statements, see Log analysis overview.
    4. Click Search & Analyze.
      In the lower section of the page, you can view the query and analysis results in a log distribution histogram and on the Raw Logs and Graph tabs. You can perform various operations based on the query and analysis results. For example, you can perform quick analysis, configure charts, and create alert rules. For more information, see Perform operations on query and analysis results and Create alert rules.
    For more information about query statements, see Query and analysis examples.

Perform operations on query and analysis results

Result type Description
Log distribution histogram The log distribution histogram is located below the search box and displays the distribution of returned log data by time. Query and analysis results
In this section, you can perform the following operations:
  • Move the pointer over a green rectangle to view the time range during which the log data is generated and the number of logs that are recorded within the time range.
  • Click a green rectangle to view the log distribution histogram within the specified time range during which the log data is generated.
Raw Logs The Raw Logs tab is located below the log distribution histogram and displays the details about each log by page. The details include the information about fields in the log. The fields are in the key:value format. Raw Logs
In Section 1 of the Raw Logs tab, you can perform the following operations:
  • Display Content Column: You can modify the display mode of the Content column in raw logs. For example, you can determine whether to display content in multiple lines, hide the default fields, expand the default levels of JSON, and fold long strings.
  • Column Settings: By default, raw logs display only the Content column. If you want to display specified fields in a column, you can click Column Settings to configure the specified fields.
  • Download: You can click the Download icon icon to download logs to your computer. Download Log in Current Page, Download All Logs with Cloud Shell, and Download All Logs Using Command Line Tool are supported. For more information, see Download logs.

In Section 2 of the Raw Logs tab, you can query logs based on fields in raw logs.

Click the value of a field in the Content column to query the logs that contain the field. For example, if you click GET of request_method: GET, and request_method: GET is automatically appended to the original query statement in the search box. This way, the system queries the results of the original query statement for the logs whose request_method is GET and returns the logs. Raw Logs

In the Quick Analysis section (Section 3) of the Raw Logs tab, you can perform the following operations:

You can analyze the distribution of a field over a specified period of time. This helps reduce the time that is required to index critical data.
  1. Click the View icon icon to the right of a field to analyze the distribution of field values. The top 10 values with the most log entries are displayed. For example, if you click the View icon icon to the right of the ua_browser field, the top 10 types of browsers are displayed.
  2. Click the Redirect icon icon to add the analytic statement that is last used to the search box. Then, you are redirected to the Graph tab on which you can view charts on analysis results.

    If the total number of values for a field exceeds 10, you can click Count Distinct Values to measure the number of distinct values.

For more information about quick analysis, see Quick analysis.
Graph The Graph tab is located below the log distribution histogram and displays the query and analysis results in charts. To view charts on the Graph tab, you must enter an analytic statement that uses the standard SQL-92 syntax in the search box. Graph tab
On the Graph tab, you can perform the following operations:
  • Change the chart type in Section 1: Select a chart type based on your business requirements to view the query and analysis results. For more information, see Chart configurations.
  • Preview a chart in Section 2: Preview the chart after you change the chart type.

    Click Add to Dashboard to add the current chart to the dashboard. Click Download Log to download logs to your computer. Download Log in Current Page, Download All Logs with Cloud Shell, and Download All Logs Using Command Line Tool are supported. For more information, see Download logs.

  • Modify the settings of a chart in Section 3:
    • On the Properties tab, you can set the properties of a chart to be displayed. You can set the X-axis, left Y-axis and right Y-axis, margins, font size, and other parameters. Different types of charts have different properties. This feature is applicable to all query scenarios.
    • On the Data Source tab, you can set placeholder variables. For example, you can configure the drill-down event of Chart A to redirect to the dashboard on which Chart B is located. After you configure the drill-down event of Chart A, the placeholder variable is replaced by the variable that you click to trigger the drill-down event and execute the query statement of Chart B. This way, to trigger the drill-down event, you must click the placeholder variable that you configured for Chart B. This feature is applicable to scenarios where you need to configure drill-down events to redirect to destination dashboards. For more information, see Configure a drill-down event.
    • On the Interactive Behavior tab, you can configure drill-down events for a chart. Then, you can click the variable value in the chart to trigger the specified drill-down event. This feature applies when you need to trigger drill-down events for charts. For more information, see Configure a drill-down event.
For more information, see Chart overview.

Create alert rules

You can create alert rules based on the current query statement. After you create an alert rule, Log Service checks related query and analysis results on a regular basis. If a query and analysis result meets the trigger condition that you specify in the alert rule, Log Service sends an alert notification. This way, the service status is monitored in real time.

To create an alert rule, you must click Save as Alert in the upper-right corner above the search box and complete the Create Alert wizard. For more information, see Configure an alert rule.

Query and analysis examples

  • Query the number of requests blocked by different WAF protection features every quarter hour. The results include the attack time (time), the numbers of requests blocked by Protection Rules Engine (wafmodule), requests blocked by the IP address blacklist and custom protection policies (aclmodule), and requests blocked by HTTP flood protection and custom protection policies (httpfloodmodule).
    * |
    SELECT
      time_series(__time__, '15m', '%H:%i', '0') as time,
      COUNT_if(final_plugin = 'waf') as "wafmodule",
      COUNT_if(final_plugin = 'acl') as "aclmodule",
      COUNT_if(final_plugin = 'cc') as "httpfloodmodule"
    GROUP by
      time
    ORDER by
      time
    The following chart displays the results. Number of requests blocked by WAF
  • Query the distribution of protection features (final_plugin) that are triggered. The results include the number of times (times) that protection features that are triggered, requested domain names (host), and protection features (final_plugin).
    * |
    SELECT
      count(*) as times,
      host,
      final_plugin
    GROUP by
      host,
      final_plugin
    ORDER by
      times desc
    The following chart displays the results. Distribution of protection features (final_plugin)
  • Query the queries per second (QPS) every quarter hour. The results include the time (time) and QPS (QPS).
    * |
    SELECT
      time_series(__time__, '15m', '%H:%i', '0') as time,
      count(*) / 900 as QPS
    GROUP by
      time
    ORDER by
      time
    The following chart displays the results. QPS query
  • Query the domain names that suffer the most HTTP flood attacks. The results include the number of times (times) that HTTP flood attacks are blocked and the targeted domain names (host).
    *
    and acl_action :block |
    SELECT
      count(*) as times,
      host
    GROUP by
      host
    ORDER by
      times desc
    The following chart displays the results. Query of HTTP flood attacks
  • Query the log details about requests every second. The results include the request time (time), accessed domain name (host), request path (request_path), request method (request_method), HTTP status code (status) that WAF responds, HTTP status code (upstream_status) that the origin server responds, and query string (querystring).
    * |
    SELECT
      date_format(date_trunc('second', __time__), '%H:%i:%s') as time,
      host,
      request_path,
      request_method,
      status,
      upstream_status,
      querystring
    LIMIT
      10
    The following chart displays the results. Request details
  • Query the latest 10 attacks on the your_domain_name website. The results include the attack time (time), actual IP address of the client, (real_client_ip), and client type (http_user_agent).
    matched_host: your_domain_name
    and final_action: block |
    SELECT
      time,
      real_client_ip,
      http_user_agent
    ORDER by
      time desc
    LIMIT
      10
    The following chart displays the results. Attack time query
  • Query the number of days (days_passed) that elapsed after an attack on the your_domain_name website was blocked by WAF. The value of days_passed is rounded to one decimal place.
    matched_host: your_domain_name
    and final_action: block |
    SELECT
      time,
      round((to_unixtime(now())-__time__) / 86400, 1) as "days_passed",
      real_client_ip,
      http_user_agent
    ORDER by
      time desc
    LIMIT
      10
    The following chart displays the results. Time query
  • Query the trend of the number of attacks on the your_domain_name website by day.
    matched_host: your_domain_name
    and final_action: block |
    SELECT
      date_trunc('day', __time__) as dt,
      count(1) as PV
    GROUP by
      dt
    ORDER by
      dt
    The date_trunc function is used to group the times when attacks occurred by day. For more information about the function, see Date and time functions.
    The following chart displays the results. We recommend that you use a line chart to display the results. Statistical analysis by group
  • Query the distribution of countries from which attacks are launched to the your_domain_name website.
    matched_host: your_domain_name
    and final_action: block |
    SELECT
      ip_to_country(
        if(real_client_ip = '-', remote_addr, real_client_ip)
      ) as country,
      count(1) as "Number of attacks"
    GROUP by
      country
    The real_client_ip field in WAF logs indicates the actual IP address of a client. If a proxy server is used or the IP field in a request header is invalid, the actual IP address of the client cannot be obtained. In this case, the value of the real_client_ip field is displayed as -. You can use the value of the remote_addr field as the actual IP address of the client. The remote_addr field indicates the IP address that is used to connect to WAF.
    The following chart displays the results. We recommend that you use the world map to display the results. World map
  • Query the distribution of provinces from which attacks are launched to the your_domain_name website.
    matched_host: your_domain_name
    and final_action: block |
    SELECT
      ip_to_province(
        if(real_client_ip = '-', remote_addr, real_client_ip)
      ) as province,
      count(1) as "Number of attacks"
    GROUP by
      province
    The ip_to_province function is used to obtain information about the provinces in which the actual IP addresses of clients are located. For more information about the function, see IP functions.
    The following chart displays the results. If all IP addresses from which attacks are launched are located inside China, we recommend that you use the China map to display the results. Distribution of visitors by province in China