Anti-DDoS Pro and Anti-DDoS Premium are integrated with Alibaba Cloud Log Service to collect and analyze website access logs, including logs of HTTP flood attacks. Log Analysis is a value-added feature. You must make a purchase to activate this feature. After you activate Log Analysis, Log Service collects access logs of the website that is protected by Anti-DDoS Pro or Anti-DDoS Premium in real time. You can query and analyze the logs, and view log reports.

Background information

The APNIC DDoS threat landscape in 2017 states that more than 80% of DDoS attacks are combined with website attacks, especially HTTP flood attacks, which are difficult to detect. Therefore, it is important to analyze access logs in real time to identify attack behavior and apply protection policies.

The Log Analysis feature of Anti-DDoS Pro or Anti-DDoS Premium is provided based on Alibaba Cloud Log Service. You can query and analyze logs on dashboards. This helps you flexibly analyze and monitor your website services. After you activate Log Analysis, you can consume and ship logs by using Log Service. This allows you to manage website access logs collected by Anti-DDoS Pro or Anti-DDoS Premium.

For more information, see What is Log Service?

Activate Log Analysis

  1. Log on to the Anti-DDoS Pro console.
  2. In the top navigation bar, select the region where your services are deployed.
    • Mainland China: Anti-DDoS Pro
    • Outside Mainland China: Anti-DDoS Premium
  3. In the left-side navigation pane, choose Investigation > Log Analysis.
  4. Click Purchase Now.Purchase Now
    If you have activated Log Analysis, the Purchase Now button does not appear. You can directly use Log Analysis. For more information, see Enable Log Analysis for a website.
  5. On the Log Service page, configure the following parameters.
    Log Service
    Parameter Description
    Applicable Product Select Anti-DDoS Pro or Anti-DDoS Premium.
    Logservice Storage Select the capacity to store logs. Unit: TB.
    Note If Log Analysis has sufficient storage capacity within the validity period, it stores logs of the last 180 days. Logs from day 181 will overwrite logs from day 1.

    Typically, each request log occupies about 2 KB of storage space. If the average QPS of your service is 500, the storage space required for a day is: 500 x 60 x 60 x 24 x 2 = 86,400,000 KB (82 GB). By default, Log Analysis stores logs over the last 180 days. Therefore, you can select a log storage capacity of 14,832 GB (14.5 TB).

    Note After the log storage capacity is exhausted, new logs cannot be stored. We recommend that you monitor the remaining log storage capacity and expand the storage capacity as needed.
    Duration Select the validity period of Log Analysis.
    Note After Log Analysis expires, new logs cannot be stored. If you do not renew Log Analysis within seven days after it expires, all log data will be automatically deleted.

    Billing

    • Log Analysis of Anti-DDoS Pro: USD 78/TB (log storage capacity) per month
    • Log Analysis of Anti-DDoS Premium: USD 156/TB (log storage capacity) per month
  6. Click Buy Now and complete the payment.
  7. Go to the Log Analysis page in the Anti-DDoS Pro or Anti-DDoS Premium console and click Authorize.
  8. On the Cloud Resource Access Authorization page, click Confirm Authorization Policy to allow Anti-DDoS Pro or Anti-DDoS Premium to store logs in the specified logstore.Cloud Resource Access Authorization
    After Log Analysis is activated, you can click Details to view details about Log Analysis.Details
    Note We recommend that you pay close attention to the remaining log storage capacity and validity period when you use Log Analysis.
    • When the usage of the log storage capacity reaches 70%, expand the capacity to make sure that new logs can be stored.
    • If a large amount of storage capacity remains unused for a long time, you can reduce the capacity.

Enable Log Analysis for a website

  1. Log on to the Anti-DDoS Pro console.
  2. In the top navigation bar, select the region where your services are deployed.
    • Mainland China: Anti-DDoS Pro
    • Outside Mainland China: Anti-DDoS Premium
  3. In the left-side navigation pane, choose Investigation > Log Analysis.
  4. On the Log Analysis page, select the domain name of the website and turn on the switch to enable Log Analysis.
    Note Before you enable Log Analysis for a domain name, you must add the domain name to Anti-DDoS Pro or Anti-DDoS Premium. The drop-down list displays the added domain names.
    Select a domain
    After Log Analysis is enabled, you can view Log Analysis and Log Reports on the Log Analysis page.Log Analysis enabled

Use Log Analysis

After you enable Log Analysis for a domain name, you can query and analyze website access logs on the Log Analysis page and view log reports provided by Anti-DDoS Pro or Anti-DDoS Premium.

Category Function Description Reference
Log Analysis You can use query and analyze logs. You can enter a query and analysis statement to query and analyze collected log data in real time. The query and analysis statement consists of a search clause and an analytics clause. The clauses are separated with a vertical bar (|).

The following example statement queries the number of visits to a website:

* | SELECT COUNT(*) as times, host GROUP by host ORDER by times desc limit 100

For more example statements, see the "Common query statements" section in this topic.

 Query and analyze logs

Fields supported by full log
Charts After you execute a query and analysis statement, the analysis results are presented in tables by default. You can select a line chart, column chart, or pie chart to present the results. Overview
Monitoring and alerts You can configure alerts based on the charts on a dashboard to monitor the service status in real time. Alert overview
Log Reports Log reports are presented on dashboards in real time. You can query and analyze logs by using statements and save the charts of analysis results to a dashboard.

Log Analysis provides two default dashboards: DDoS Access Center and DDoS Operation Center.

You can also subscribe to a dashboard or send the dashboard to specific recipients by using emails or DingTalk messages.

Query log reports

Scenarios

Log Analysis can meet the following requirements:
  • Troubleshoot website access issues.

    After Log Analysis is enabled for your domain name, you can query and analyze logs collected from your website in real time. You can use SQL statements to analyze website access logs. This allows you to troubleshoot and analyze access issues, and view information about read/write latency and the distribution of ISPs.

    The following example statement queries access logs of your website:

    __topic__: DDoS_access_log

  • Track HTTP flood attacks.
    Access logs record sources and distribution of HTTP flood attacks. You can query and analyze access logs in real time to identify the attack sources and implement protection strategies.
    • The following example statement analyzes geographical distribution of HTTP flood attacks:

      __topic__: DDoS_access_log and cc_blocks > 0| SELECT ip_to_country(if(real_client_ip='-', remote_addr, real_client_ip)) as country, count(1) as "Number of attacks" group by country

    • The following example statement queries PVs of your website:

      __topic__: DDoS_access_log | select count(1) as PV

  • Analyze website operations.

    Access logs record information about website traffic in real time. You can use SQL statements to analyze logs and obtain information about website operations in real time. For example, you can identify the most visited webpages, source IP addresses of the clients, the browsers that initiated the requests, and the distribution of clients.

    The following example statement queries the traffic distribution from various ISPs:

    __topic__: DDoS_access_log | select ip_to_provider(if(real_client_ip='-', remote_addr, real_client_ip)) as provider, round(sum(request_length)/1024.0/1024.0, 3) as mb_in group by provider having ip_to_provider(if(real_client_ip='-', remote_addr, real_client_ip)) <> '' order by mb_in desc limit 10

Common query statements

  • Query types of blocked requests. 
    * | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
  • Query the QPS. 
    * | select time_series(__time__,'15m','%H:%i','0') as time,count(*)/900 as QPS group by time order by time
  • Query attacked domain names. 
    * and cc_blocks:1 | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
  • Query attacked URLs. 
    * and cc_blocks:1 | select count(*) as times,host,request_path group by host,request_path order by times
  • Query request details. 
    * | select date_format(date_trunc('second',__time__),'%H:%i:%s') as time,host,request_uri,request_method,status,upstream_status,querystring limit 10
  • Query 5XX HTTP status codes. 
    * and status>499 | select host,status,upstream_status,count(*)as t group by host,status,upstream_status order by t desc
  • Query the distribution of request latency. 
    * | SELECT count_if(upstream_response_time<20) as "<20",
count_if(upstream_response_time<50 and upstream_response_time>20) as "<50",
count_if(upstream_response_time<100 and upstream_response_time>50) as "<100",
count_if(upstream_response_time<500 and upstream_response_time>100) as "<500",
count_if(upstream_response_time<1000 and upstream_response_time>500) as "<1000",
count_if(upstream_response_time>1000) as ">1000"