Troubleshoot traffic anomalies by analyzing logs - Cloud Firewall

By querying and analyzing collected log data in real time, you can accurately understand your network activity, identify traffic anomalies, enhance security monitoring, and improve incident response. This topic describes how to use search statements and analytic statements in Log Analysis, and how to interpret the query results.

Statement syntax

Search statements and analytic statements

Log Analysis statements consist of a search statement and an analytic statement, separated by a vertical bar (|).

search statement|analytic statement

A search statement uses proprietary Simple Log Service syntax to retrieve logs that meet specific conditions. You can use a search statement on its own. A search statement can contain keywords, numbers, numeric ranges, spaces, or an asterisk (*). If the statement consists only of a space or an asterisk (*), no filter is applied.
Based on how your index is configured, you can perform a full-text search or a field-specific search. For more information, see Query syntax and functions.
Full-text search
To perform a full-text search, specify one or more keywords. Enclose a keyword in double quotation marks ("") for an exact match. To find logs that contain all specified keywords, separate the keywords with spaces or the and operator.
- Multi-keyword search
  Search for logs that contain both www.aliyun.com and pass.
```
www.aliyun.com pass
```
  or
```
www.aliyun.com and pass
```
- Conditional search
  Search for all logs that contain www.aliyun.com and either pass or tcp.
```
www.aliyun.com and (pass or tcp)
```
- Prefix search
  Search for all logs that contain www.aliyun.com and start with tcp_.
```
www.aliyun.com and tcp_*
```
  Note
  In a prefix search, you can only use the asterisk (*) as a suffix. Using * as a prefix, such as *_not_establish, is not supported.
Field-specific search
To perform a field-specific search, specify a field name and value. A field-specific search supports comparison operators for numeric fields, using formats like field:value or field >= value. You can also combine multiple conditions by using operators such as and and or, and combine a field-specific search with a full-text search.
For a list of fields that support indexing in Cloud Firewall Log Analysis, see Fields that support indexes.
- Multi-field search
  Query access logs for traffic from the client 192.XX.XX.22 to the destination address 192.XX.XX.54.
```
src_ip: 192.XX.XX.22 and dst_ip: 192.XX.XX.54
```
- Search based on field existence
  - Query for logs that contain the cloud_instance_id field.
    cloud_instance_id: *
  - Query for logs that do not contain the cloud_instance_id field.
    not cloud_instance_id: *
An analytic statement performs statistical calculations on search results. It must be used with a search statement. If the analytic statement is omitted, no analysis is performed and only the search results are returned. For more information about the syntax and functions for analytic statements, see Overview of log query and analysis.

Common query statements

This section provides common query statements and examples for Cloud Firewall logs.

Traffic volume

Query network records for inbound internet traffic to internal assets with the IP address prefix 1.2.*.*, and calculate the total inbound traffic volume and packet count.
```
log_type:internet_log and direction:"in" and dst_ip:1.2.*.* | select sum(in_packet_bytes) as flow, sum(in_packet_count) as packet
```

Query traffic on the NAT boundary firewall, group the results by source IP, destination IP, and destination port, and then analyze the top 10 assets by total inbound and outbound traffic.

log_type:nat_firewall_log | select src_ip, dst_ip, dst_port, sum(in_packet_bytes) as in_bytes, sum(out_packet_bytes) as out_bytes, sum(total_packet_bytes) as total_bytes group by src_ip, dst_ip, dst_port order by total_bytes desc limit 10

Access control

Query traffic logs for inbound internet traffic to internal assets that matched an access control policy.
```
log_type:internet_log and direction:"in" and not acl_rule_id:00000000-0000-0000-0000-000000000000
```
A value of 0 for not acl_rule_id indicates that an access control policy is matched. Any other value indicates that an access control policy is not matched.

Query traffic logs for outbound internet traffic from internal assets that was intercepted by an access control policy, and then analyze the distribution of the top 10 destination IP addresses and ports.

log_type:internet_log and direction:out and not acl_rule_id:00000000-0000-0000-0000-000000000000 and rule_result:drop | select dst_ip, dst_port, count(*) as cnt group by dst_ip, dst_port order by cnt desc limit 10

Query traffic logs for outbound internet traffic from internal assets under the following conditions: the destination port is 443, the packet count is greater than 3 (indicating a complete TCP three-way handshake), and the domain is unknown. Then, group the results by destination IP subnet and application, and analyze the top 10 source IPs, destination IPs, and applications.
```
log_type:internet_log and direction:out and dst_port:443 and total_packet_count>3 and domain:""| select array_agg(distinct src_ip) as srcip, array_agg(distinct dst_ip) as dstip, slice(split(dst_ip,'.' ,4),1,3) as dstip_c, app_name, COUNT(1) as cnt GROUP by dstip_c,app_name order by cnt desc limit 10
```

Attack prevention

Query traffic logs for inbound internet traffic to internal assets that matched an attack prevention policy.
```
log_type:internet_log and direction:"in" and not ips_rule_id:00000000-0000-0000-0000-000000000000
```
A value of 0 for not ips_rule_id indicates that an attack prevention policy is matched. Otherwise, the policy is not matched.
Query traffic logs for outbound internet traffic from internal assets that matched an attack prevention policy, and view information such as IP addresses, ports, applications, domains, and IPS policy results.
```
log_type:internet_log and direction:out and not ips_rule_id:00000000-0000-0000-0000-000000000000 | select src_ip, dst_ip, dst_port,app_name, domain,ips_rule_id, ips_rule_name, rule_result
```

Procedure

Log on to the Cloud Firewall console.
In the left-side navigation pane, choose Detection & Response > Log Analysis, and then click the Logs tab.
(Optional) By default, when you navigate to the Logs tab, the system automatically performs a query and displays the results. You can click the icon in the upper-right corner and, on the Query Settings tab, disable this feature or set a query time.
In the search box, enter a search statement and an analytic statement.
- A search statement filters log data based on criteria such as a time range or keyword. A search statement can be used on its own. For more information about the syntax, see Query syntax and functions.
- An analytic statement filters, transforms, and aggregates log data. For example, you can calculate average values or compare data across different time periods. An analytic statement must be used with a search statement in the format search statement|analytic statement. For more information about the syntax, see Aggregate functions.
You can set the time range for the search & analyze operation in one of three ways. If a time range is also specified in the analytic statement, it takes precedence.
Note
By default, a query and analysis statement returns only 100 rows. To return more data, use the LIMIT clause. For more information, see LIMIT clause.
- From the drop-down list at the top of the page, select a time range, such as 15 Minutes.
- In the analytic statement, use the __time__ field to specify a time range (inclusive), for example:
```
* | SELECT * FROM log WHERE __time__>1731297600 AND __time__< 1731310038
```
- When you specify a time in an analytic statement, use the from_unixtime function or to_unixtime function to convert the time format. For example:
  - ```
  * | SELECT * FROM log WHERE from_unixtime(__time__) > from_unixtime(1731297600) AND from_unixtime(__time__) < now()
```
- ```
* | SELECT * FROM log WHERE __time__ > to_unixtime(date_parse('2024-10-19 15:46:05', '%Y-%m-%d %H:%i:%s')) AND __time__ < to_unixtime(now())
```
Click search & analyze to view the results. For details, see View query and analysis results.

View query and analysis results

You can view your query results in modules such as the histogram and raw logs.

Note

If an analytic statement does not include a LIMIT clause, a maximum of 100 results are returned by default. To return more results, you must specify a LIMIT value.
The following sections describe the data displayed in the histogram and raw logs. For information about other features, see Quick start for query and analysis.

Histogram

The histogram displays the time distribution of the queried logs.

Hover over a green bar to view the time range and the number of log entries it represents.
Double-click a green bar to view the log distribution in a more granular time frame. The Raw Logs tab will update to show the query results for that specific time range.

Raw logs

The Raw Logs tab displays your query results.

Number	Description
1	Switch between log display formats, change the time sort order, and more.
2	Click the icon to download logs to your local machine or view your download history. For more information, see Export logs.
3	Click the icon and select JSON Configuration to set the JSON expansion type and level. Click the icon and select Event Configuration to configure log drill-down features. This feature helps you visually drill down into raw logs for more details. For more information, see Event Configuration.
4	Displays display fields, index fields, and system fields. In the index fields area, click the icon next to a field to add it to the display fields. The field will then appear in the log details on the right. In the display fields area, click the icon next to a field to remove it from the display fields. The field will no longer appear in the log details on the right. Note If no display fields are added, the log details on the right show the system's default fields. Click the icon next to a field to view its Basic Distribution, Statistical Metrics, and other information. For more information, see Field Settings.
5	After you configure the display fields, click the icon to save the list of display fields as a view, so you can quickly switch to this set of fields later. Click the icon and select Tag Settings to set a field as a system tag. The tag is displayed above the log details on the right. For example: . Click the icon and enable the Alias feature to display aliases for field names. Fields without an alias continue to show their original names. For steps on how to set a field alias, see Create an index.
6	Displays the log details. For details about log fields, see Log fields. Click the icon to copy the log content. Click the icon to use SLS Copilot to summarize the log or find errors. Click the icon to view tag details.

Cloud Firewall:Query and analyze logs