After you enable the log analysis feature in the Cloud Firewall console, you can perform operations, such as querying and analyzing collected logs in real time, viewing or editing dashboards, and configuring alert rules for monitoring.

Prerequisites

internet_log and vpc_firewall_log that are displayed in the upper-right corner of the Log Analysis page are turned on.

Procedure

  1. Log on to the Cloud Firewall console.
  2. In the left-side navigation pane, choose Log Analysis > Log Analysis.
  3. On the Log Analysis page, click the Logs tab.
  4. Enter a query statement, specify a time range, and then click Search & Analyze.

    Log Service supports a wide range of query statements that can be used in complex scenarios. For more information, see Use custom statements to query and analyze logs.

    Query logs

Perform operations on the Log Analysis page

On the Log Analysis page, you can perform the following operations:
  • View the distribution of logs within a specific time range
    The column chart below the search box shows the distribution of logs that are queried by using a specific query statement within a specified time range. The x-axis indicates the time range, and the y-axis indicates the number of log entries. The total number of log entries is displayed below the chart.
    Note You can drag the pointer on the histogram to narrow down the time range. The time picker automatically changes the time range, and the search results are updated.
    The distribution of logs within a specific time range
  • View raw logs

    The Raw Logs tab shows the details of each log entry, including the time, content, and fields. You can click Display Content Column and set Key-Value Pair Arrangement to Full Line or New Line based on your business requirements. You can click Column Settings to select the columns to display. You can also click the Download icon icon to download the search results to your computer.

    If you click a numeric field value or a field value that is tokenized in the Content column, the field and its value are automatically inserted into the search box as a query condition. For example, click log_service in __source__: log_service. The query statement is changed to the following statement:

    <Original statement> and source: log_service
    Raw logs
  • View charts of log analysis results

    Log analysis results are displayed in charts. Various types of charts are provided on the Graph tab. You can select one based on your requirements. For more information, see Overview.

    Statistical chart
  • Perform quick analysis

    On the Raw Logs tab, you can view the distribution of a log field within a specific time range with a few clicks. This helps reduce the time required to query key data. For more information, see Quick analysis.

    Quick Analysis

Use custom statements to query and analyze logs

A query statement consists of a search statement and an analytic statement that are separated by a vertical bar (|).

$Search | $Analytics
Type Description
Search statement A search statement uses syntax that is specific to log analysis and is used to query the logs that meet specific search conditions. Search conditions include keywords, fuzzy strings, numeric values, ranges, or combinations of these items. If the statement is empty or contains only an asterisk (*), all logs are queried.
Analytic statement An analytic statement uses the SQL-92 syntax and is used to calculate and analyze search results. If the statement is empty, only search results are returned but no calculations are performed on the results.

Search syntax

The search syntax of Log Service supports both full-text query and field-based query. The search box supports features such as multi-line search and syntax highlighting.
  • Full-text query

    When you enter a keyword to query logs, you do not need to specify fields. If you want to query the logs that contain the keyword, you can enclose the keyword in a pair of double quotation marks (""). If you enter multiple keywords, separate them with spaces or combine them by using and.

    Examples
    • Query logs based on multiple keywords

      You can query the logs that contain both www.aliyun.com and error.

      www.aliyun.com error or www.aliyun.com and error

    • Query logs based on a condition

      You can query the logs that contain www.aliyun.com and contain error or 404.

      www.aliyun.com and (error or 404)
    • Query logs based on a prefix

      You can query the logs that contain www.aliyun.com and start with failed_.

      www.aliyun.com and failed_*
      Note The asterisk ( *) can be added only as a suffix. It cannot be added as a prefix. For example, *_error is not supported.
  • Field-based query

    You can query logs based on fields.

    You can specify a numeric field and value in the format of Field name: Value or Field name >= Value. You can also use operators to specify a combination of fields. The operators include and and or. Furthermore, you can use field-based query together with full-text query.
    Note The log analysis feature of Cloud Firewall allows you to perform field-based query to obtain logs. For information about the definition, type, and format of each field, see Log fields.
    Examples
    • Query logs based on multiple fields

      If you want to query the logs that record access requests from the client whose IP address is 192.0.2.0 to 192.0.2.54, use the following search condition:

      src_ip: 192.0.2.0 and dst_ip: 192.0.2.54
      Note In the example, src_ip and dst_ip are log fields that are recorded by Cloud Firewall.
    • Query logs based on field existence
      • You can query the logs that contain the total_pps field.
        total_pps: *
      • You can query the logs that do not contain the total_pps field.
        not total_pps: *
For more information about the query statements supported by Log Service, see Overview.

Analytics syntax

You can execute SQL-92 statements to analyze logs.

For more information about the statement syntax and functions supported by Log Service, see Real-time log analysis.
Note
  • You can omit the from Table name clause in standard SQL statements. This clause is equivalent to from log.
  • By default, the first 100 log entries are returned. If you want to adjust this number, you can execute a LIMIT statement. For more information, see LIMIT syntax.

Examples of log query and analysis

Time-based log query and analysis

Each log entry has a time field, which presents the time when the log entry was generated in the format of YYYY-MM-DDThh:mm:ss+Time zone. For example, in 2018-05-31T20:11:58+08:00, the time zone is UTC+8.

Each log entry has a built-in field __time__. This field also indicates the time when the log entry is generated. The time is a UNIX timestamp and is used in time-based calculation. The value of this field indicates the number of seconds that have elapsed since the UTC time 00:00:00, January 1, 1970. If you want to display recognizable calculation results, you must convert the format first. For more information about time resolving functions, such as date_parse and date_format that are used to convert a time format to another, see Date and time functions.