This topic describes how to enable and use the Log Analysis feature provided by Anti-DDoS Pro or Anti-DDoS Premium.

Prerequisites

  • An Anti-DDoS Pro or Anti-DDoS Premium instance is purchased and your website is added to Anti-DDoS Pro or Anti-DDoS Premium. For more information, see Add a website.

    Before you can use Log Analysis to collect and store the logs of your website, and then query and analyze the collected logs, you must add the website to Anti-DDoS Pro or Anti-DDoS Premium.

  • Log Service is activated.

    If this is the first time you log on to the Log Service console, you must activate Log Service as prompted.

Step 1: Enable Log Analysis

Perform the following steps to enable Log Analysis:

  1. Log on to the Anti-DDoS Pro console.
  2. In the top navigation bar, select the region where your instance resides.
    • Mainland China: If you select this region, the Anti-DDoS Pro console appears.
    • Outside Mainland China: If you select this region, the Anti-DDoS Premium console appears.
    You can switch the region to configure and manage Anti-DDoS Pro or Anti-DDoS Premium instances. Make sure that you select the required region when you use Anti-DDoS Pro or Anti-DDoS Premium.
  3. In the left-side navigation pane, choose Investigation > Log Analysis.
  4. On the Log Analysis page, click Purchase Now.
    If you have enabled Log Analysis, Purchase Now does not appear. You can directly use the feature. For more information, see Step 2: Enable the log collection feature. Purchase Now
  5. On the Log Service page, configure the following parameters.
    Log Service
    Parameter Description
    Applicable Product Select Anti-DDoS Pro or Anti-DDoS Premium.
    Logservice Storage Select the capacity to store logs. Unit: TB. Valid values: 3T, 5T, 10T, 20T, 50T, 100T, and 1000T.

    If log storage is large enough and within the validity period, logs are stored from the first day the feature is used. The logs that are generated within the following 180 consecutive days are stored. Logs from day 181 overwrite logs from day 1, which indicates that the logs generated only within the last 180 days are stored.

    In most cases, each request log occupies about 2 KB of storage. If the average queries per second (QPS) of your service is 500, the storage required for a day is 86,400,000 KB (about 82 GB). The storage is calculated based on the following formula: 500 x 60 x 60 x 24 x 2 = 86,400,000. If you want to store logs of the last 180 days, the storage required is 14,832 GB (about 14.5 TB), and you need to specify the Logservice Storage parameter based on this value. The default log retention period is 180 days.

    Notice After the log storage is exhausted, new logs cannot be stored.
    Duration Select a validity period for the feature. Valid values: 1 Month, 2 Months, 3 Months, 6 Months, 1 Year, and 2 Years.
    Notice If Log Analysis expires, new logs cannot be stored.
  6. Click Buy Now and complete the payment.
    After you purchase Log Analysis, Log Service automatically creates a dedicated project for Anti-DDoS Pro or Anti-DDoS Premium. This dedicated project is used to manage the logs of Anti-DDoS Pro or Anti-DDoS Premium. You can view the dedicated project on the homepage of the Log Service console. Log Project

    The name of the dedicated project for Anti-DDoS Pro starts with ddoscoo-project. The name of the dedicated project for Anti-DDoS Premium starts with ddosdip-project.

    A dedicated project for Anti-DDoS Pro or Anti-DDoS Premium contains the following resources:
    • A dedicated Logstore that is used to store the logs of Anti-DDoS Pro or Anti-DDoS Premium. The name of the dedicated Logstore for Anti-DDoS Pro starts with ddoscoo-logstore. The name of the dedicated Logstore for Anti-DDoS Premium starts with ddosdip-logstore.
    • Two preset log dashboards that are used to display the log analysis results in charts. The dashboards are DDoS Access Center and DDoS Operation Center. The information in the dashboards is the same for both Anti-DDoS Pro and Anti-DDoS Premium.
  7. Go to the Log Analysis page and authorize Anti-DDoS Pro or Anti-DDoS Premium to store logs to the dedicated Logstore of Log Service.
    Note You need to perform the authorization operations only once. If you have completed the authorization, skip this step.
    Perform the following steps to authorize Anti-DDoS Pro or Anti-DDoS Premium:
    1. Click Authorize.
    2. On the Cloud Resource Access Authorization page, click Confirm Authorization Policy. Cloud Resource Access Authorization page

After you enable the Log Analysis feature and complete authorization, you can start to use this feature on the Log Analysis page. Before you use this feature, you must enable log collection for the domain name of your website. For more information, see Step 2: Enable the log collection feature.

Step 2: Enable the log collection feature

By default, Anti-DDoS Pro and Anti-DDoS Premium do not collect logs of the added websites. Anti-DDoS Pro and Anti-DDoS Premium collect the logs of the websites and store the collected logs to the dedicated Logstores in Log Service only after you enable log collection for the domain names of the websites. Then, you can query and analyze the logs.

Perform the following steps to enable log collection for the domain name of a website:

  1. Log on to the Anti-DDoS Pro console.
  2. In the top navigation bar, select the region where your instance resides.
    • Mainland China: If you select this region, the Anti-DDoS Pro console appears.
    • Outside Mainland China: If you select this region, the Anti-DDoS Premium console appears.
    You can switch the region to configure and manage Anti-DDoS Pro or Anti-DDoS Premium instances. Make sure that you select the required region when you use Anti-DDoS Pro or Anti-DDoS Premium.
  3. In the left-side navigation pane, choose Investigation > Log Analysis.
  4. On the Log Analysis page, enable log collection for the domain name of a website.
    Notice Before you enable log collection for a domain name, you must add the domain name to Anti-DDoS Pro or Anti-DDoS Premium. The domain name list displays the added domain names.
    You can use one of the following methods to enable log collection for domain names:
    • Enable log collection for a domain name: Select a domain name from the Select a domain drop-down list and turn on Status.
    • Enable log collection for multiple domain names at a time: Click Batch config in the upper-right corner of the page. In the Batch config panel, select multiple domain names and click Turn on in a batch.
    Select domain names
    After log collection is enabled, Anti-DDoS Pro or Anti-DDoS Premium collects and stores the logs of websites for query and analysis. For more information about how to query and analyze logs, see Step 3: Use Log Analysis.

Step 3: Use Log Analysis

After you enable log collection for a domain name, you can query and analyze the collected logs on the Log Analysis tab of the Log Analysis page. You can also view the log reports in the dashboards that are preset for Anti-DDoS Pro or Anti-DDoS Premium on the Log Reports tab. Log Analysis and Log Reports tabs

The following table describes the features that are provided on the Log Analysis page. For more information, see Common operations on logs of Alibaba Cloud services.

Tab Feature Description References
Log Analysis Log query and analysis You can query and analyze the collected log data in real time. A query and analysis statement consists of a search clause and an analytics clause that are separated by a vertical bar (|).

For example, you can use the following statement to query the number of visits to a domain:

* | SELECT COUNT(*) as times, host GROUP by host ORDER by times desc limit 100

For more information about query and analysis statements, see Common query statements.

Query and analyze logs

Fields supported by full log

Analysis results in charts A query and analysis statement contains the syntax for analytics. After the statement is executed, analysis results are automatically displayed in tables. The analysis results can also be displayed in a variety of charts, such as a line chart, column chart, or pie chart. You can choose a display method based on your business requirements. Chart overview
Monitoring and alerting You can configure alert rules based on the charts in a dashboard to monitor service status in real time. Alerting overview
Log Reports Dashboard Log Service provides dashboards for you to analyze data in real time. After you query and analyze logs by using query and analysis statements, you can save the charts of analysis results to a dashboard. Log Analysis provides two preset dashboards: DDoS Access Center and DDoS Operation Center.

You can also subscribe to dashboards and send dashboard data to specific recipients by using emails or DingTalk messages.

Query log reports

Subscribe to a dashboard

Step 4: Manage the configurations

The specifications of the Log Analysis feature are displayed in the upper-right corner of the Log Analysis page. You can perform the following operations in this section: Log Analysis
  • Query the validity period of the Log Analysis feature. If Log Analysis is about to expire, you can click Renew to extend the validity period of the feature.
    Warning If Log Analysis expires, new logs cannot be stored. Seven days after Log Analysis expires, all existing logs are cleared.
  • Query the usage of log storage. If log storage is to be exhausted, you can click Upgrade to expand log storage. Alternatively, you can click Clear to delete the logs that are no longer required.

    The usage of log storage displayed in the Anti-DDoS Pro console is not updated in real time. The displayed usage does not include the usage from the last two hours.

    Note We recommend that you check the usage of log storage at regular intervals when you use Log Analysis. When the usage of log storage exceeds 70%, expand the log storage to make sure that new logs can be stored. If a specific amount of log storage remains idle for a long period of time, you can reduce the log storage.
  • Change the duration to store logs. Logs are stored for 180 days by default. You can click Details and set Storage Period to a value that ranges from 30 to 180 in the Details dialog box. Unit: days. Details

Common query statements

  • Queries the type of attacks that are blocked.
    * | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
  • Queries the QPS.
    * | select time_series(__time__,'15m','%H:%i','0') as time,count(*)/900 as QPS group by time order by time
  • Queries the domain names that are attacked.
    * and cc_blocks:1 | select cc_action,cc_phase,count(*) as t group by cc_action,cc_phase order by t desc limit 10
  • Queries the URLs that are attacked.
    * and cc_blocks:1 | select count(*) as times,host,request_path group by host,request_path order by times
  • Queries the details about a request.
    * | select date_format(date_trunc('second',__time__),'%H:%i:%s') as time,host,request_uri,request_method,status,upstream_status,querystring limit 10
  • Queries the details about the 5XX status codes.
    * and status>499 | select host,status,upstream_status,count(*)as t group by host,status,upstream_status order by t desc
  • Queries the distribution of request latencies.
    * | SELECT count_if(upstream_response_time<20) as "<20",
    count_if(upstream_response_time<50 and upstream_response_time>20) as "<50",
    count_if(upstream_response_time<100 and upstream_response_time>50) as "<100",
    count_if(upstream_response_time<500 and upstream_response_time>100) as "<500",
    count_if(upstream_response_time<1000 and upstream_response_time>500) as "<1000",
    count_if(upstream_response_time>1000) as ">1000"