Analyze SAG traffic with Simple Log Service

You can use Simple Log Service to query and analyze traffic from a Smart Access Gateway (SAG) instance, helping you understand traffic distribution and troubleshoot network issues.

Prerequisites

You have created a project and a logstore in Simple Log Service. For more information, see Quick start for Simple Log Service.
Your SAG device is connected to Alibaba Cloud. For more information, see Connect an SAG device to Alibaba Cloud by using static routing.
Make sure that your SAG device model is SAG-1000.

Background

Smart Access Gateway (SAG) supports the flow log feature. A flow log captures IP traffic information that passes through your SAG instances and stores the data in Alibaba Cloud Simple Log Service or on a NetFlow collector. The following procedure describes how to use Simple Log Service to store, query, and analyze this traffic.

Step 1: Ingest data

To query and analyze traffic, you must first record the traffic information from your SAG instance in a specified project and logstore.

Create a flow log.

Create a flow log in the SAG console to store traffic information in a specific project and logstore.
1. Log on to the SAG console.
2. In the left-side navigation pane, click Flow Log.
3. On the Flow Log page, click Create Flow Log.
4. In the Create Flow Log panel, configure the following parameters and then click OK.
  - Name: Enter a name for the flow log.
  - Output Interval Under Active Connections: Specify the interval at which to output log data for active network connections. The default value is 300 seconds. The value must be in the range of 60 to 6,000 seconds.
  - Output Interval Under Inactive Connections: Specify the interval at which to output log data for inactive network connections. The default value is 15 seconds. The value must be in the range of 10 to 600 seconds.
  - Deliver Flow Log Data To: Select a storage type for the logs. For this topic, select SLS.
    - To store the log data in Simple Log Service, select SLS.
    - To store the log data on your NetFlow collector, select NetFlow.
    - To store the log data in both Simple Log Service and your NetFlow collector, select ALL.
  - SLS Region: Select the region where Simple Log Service is deployed.
  - SLS Project: Select the project that contains the logstore used to store flow log data.
  - SLS Logstore: Select the logstore to store flow log data.
  For more information about the parameters, see Create a flow log.
Associate the flow log with an SAG instance.

After creating the flow log, associate it with an SAG instance so that the traffic information is stored in the specified project and logstore for querying and analysis.
1. On the Flow Log page, find the flow log that you created and click its ID.
2. On the details page of the flow log, click Add Instance.
3. In the Add Instance panel, select the target SAG instance and then click Save.

Step 2: Query and analyze traffic

After you configure the flow log, query and analyze the collected SAG traffic in the Simple Log Service console.

Log on to the Simple Log Service console.
In the Projects section, click the target project.
On the Log Storage > Logstore Name tab, click the target logstore.
Enable indexing. For more information, see Enable and configure an index.

An index is a storage structure that sorts data from one or more columns. You must configure an index before you can query and analyze logs. In this example, enable field indexing and turn on the statistics feature. In the Field Search section, configure field indexing for the following fields: ali_uid (long type), bytes, dstaddr, dstport, instance_id, packets, protocol, snid, srcaddr, srcport, tos (text type), end, start (double type), and tcp-flags (text type). For text type fields, use ,";=()[]{}?@&<>/:\n as delimiters. For the ali_uid, bytes, dstaddr, dstport, packets, protocol, srcaddr, srcport, and tos fields, enable statistics.

Note
When you configure field indexing, ensure the bytes field is the TEXT type to enable data analysis and statistics.
After enabling indexing, you can query and analyze the data. The following example shows how to query for the top 10 5-tuples.
1. In the search box, enter a query statement.
  The fields that correspond to 5-tuple traffic are srcaddr, srcport, dstaddr, dstport, and protocol.
```
* | select srcaddr,srcport,dstaddr,dstport,protocol,count(*) as num,sum(bytes) as bytes
from (select CASE
WHEN strpos(bytes, 'M') != 0 then
(CAST(replace(bytes,'M') AS double)*1024*1024)
WHEN strpos(bytes, 'K') != 0 then
(CAST(replace(bytes,'K') AS double)*1024)
else CAST(bytes AS double) end
as bytes,srcaddr,srcport,dstaddr,dstport,protocol from log limit 100000)
GROUP BY srcaddr,dstaddr,srcport,dstport,protocol ORDER BY bytes DESC limit 10
                                
```
  By default, the system searches for data in the last 15 minutes. You can specify a custom time range for your query. Enter the preceding SQL statement in the query editor, set the time range to 15 Minutes (Relative), and then click Search & Analyze to run the query.
  
  Note
  When you write a query statement, make sure that the fields are consistent with those in your logs. The fields used in this topic are for reference only. For more information about query statements, see Query overview.
2. Click Search & Analyze.
  
  The console opens the Graph tab and displays the statistics for the top 10 5-tuples in a table. You can select other chart types to visualize the data. For more information, see Chart overview.
  
  This example uses a pie chart to present the statistical data.
3. On the Graph tab, adjust the pie chart properties to customize the chart.
  Adjust the following two properties. Default values are used for other properties. For more information, see Pie charts.
  - Category: The data category.
    
    In this example, the data is categorized by the srcaddr, srcport, dstaddr, dstport, and protocol fields. A record is counted only when all five fields of the traffic match.
  - Value Column: The numerical value that corresponds to the categorized data.
    
    In this example, the bytes field is used as the value column.
Optional: You can perform the preceding operations to query the top 10 3-tuple information and the top 10 source-destination IP address information.
- Query the top 10 3-tuple information
  - Query fields: srcaddr, dstaddr, and protocol.
  - Query statement:
```
* | select srcaddr,dstaddr,protocol,count(*) as num,sum(bytes) as bytes
from (select CASE
WHEN strpos(bytes, 'M') != 0 then
(CAST(replace(bytes,'M') AS double)*1024*1024)
WHEN strpos(bytes, 'K') != 0 then
(CAST(replace(bytes,'K') AS double)*1024)
else CAST(bytes AS double) end
as bytes, srcaddr,dstaddr,protocol from log limit 100000)
GROUP BY srcaddr,dstaddr,protocol ORDER BY bytes DESC limit 10
                                        
```
  - Query result: Select the Graph tab. In the Properties pane on the right, set Chart Type to Pie Chart, select srcaddr, dstaddr, and protocol for Category, and select bytes for Value Column. Turn on Show Legend and set Legend Position to Right. For formatting, select K,Mil,Bil. For Scale Label Format, select Percentage. The preview area on the left displays a pie chart of the traffic distribution by source address, destination address, and protocol, along with a data preview table.
- Query the top 10 source-destination IP address information
  - Query fields: srcaddr and dstaddr.
  - Query statement:
```
* | select srcaddr,dstaddr,count(*) as num,sum(bytes) as bytes
from (select CASE
WHEN strpos(bytes, 'M') != 0 then
(CAST(replace(bytes,'M') AS double)*1024*1024)
WHEN strpos(bytes, 'K') != 0 then
(CAST(replace(bytes,'K') AS double)*1024)
else CAST(bytes AS double) end
as bytes, srcaddr,dstaddr from log limit 100000)
GROUP BY srcaddr,dstaddr ORDER BY bytes DESC limit 10
```
  - Query result: After you run the query statement, click the Graph tab. In the Properties pane on the right, set Chart Type to Pie Chart. For Category, select srcaddr and dstaddr. For Value Column, select bytes. The pie chart shows the proportion of traffic volume by source and destination address. You can click Add to Dashboard to save the chart.

Step 3: (Optional) Add a chart to a dashboard

You can save query and analysis results as charts on a dashboard for repeated viewing.

In the upper-right corner of the pie chart, click Add to Dashboard.
In the Add to Dashboard dialog box, configure the following parameters and then click OK.
- Operation: Select Create Dashboard.
- Dashboard Name: Enter a name for the dashboard, such as 5-tuple statistics.
- Chart Name: Enter a name for the chart, such as 5-tuple pie chart statistics.
For more information, see Add a chart to a dashboard.
In the left-side navigation pane, click Dashboard.
Click the name of the dashboard that you created to view the dashboard.

On the dashboard, click Time Range to view query and analysis results for different periods. For more information, see Display mode. The 5-tuple pie chart statistics chart is displayed on this dashboard. You can use the Time Range button to change the time range for the statistics.