Virtual Private Cloud:Flow logs

Console

API

Before creating a flow log, make sure you have activated the flow log feature and created a project and a Logstore in SLS:

• Call OpenFlowLogService to enable the flow log feature.

• Call CreateProject to create a project, and call CreateLogStore to create a Logstore.

After you meet these prerequisites, you can perform the following operations:

• Call CreateFlowLog to create a flow log, and optionally call CreateIndex to create an index.

• Call DeactiveFlowLog to stop a flow log.

• Call ActiveFlowLog to start a flow log.

• Call DeleteFlowLog to delete a flow log.

Terraform

Resources: alicloud_log_project, alicloud_log_store, alicloud_vpc_flow_log

# Specify the region where you want to create the flow log.
provider "alicloud" {
  region = "cn-hangzhou"
}

# Specify the description of the project, and the names of the Logstore and flow log.
variable "name" {
  default = "vpc-flowlog-example"
}

# Generate a random number to create a project name.
resource "random_uuid" "example" {
}

# Create a Simple Log Service project.
resource "alicloud_log_project" "example" {
  project_name = substr("tf-example-${replace(random_uuid.example.result, "-", "")}", 0, 16)
  description  = var.name
}

# Create a Simple Log Service Logstore.
resource "alicloud_log_store" "example" {
  project_name          = alicloud_log_project.example.project_name
  logstore_name         = var.name
  shard_count           = 3
  auto_split            = true
  max_split_shard_count = 60
  append_meta           = true
}

# Create a VPC flow log.
resource "alicloud_vpc_flow_log" "example" {
  flow_log_name        = var.name
  log_store_name       = alicloud_log_store.example.logstore_name
  description          = var.name
  traffic_path         = ["all"] # Collect traffic from all scenarios.
  project_name         = alicloud_log_project.example.project_name
  resource_type        = "VPC" # The resource type is VPC.
  resource_id          = "vpc-bp1ekmgzch0bo3hxXXXXXX" # The ID of the VPC.
  aggregation_interval = "1" # The capture window is 1 minute.
  traffic_type         = "All" # Collect all traffic, regardless of whether it is allowed or rejected by access control.
}

Console

API

Call GetLogsV2 to query and analyze flow logs.

Source IPs of public access to ECS instances

As shown in the figure, assume that you have created a web server that is accessible from the Internet over port 80 and you have used security group rules to restrict access from source IPs.

You can create a flow log to query the source IPs that access port 80 and count the access requests that are allowed or rejected by the security group.

1. Create a flow log

   • For Resource Instance, select the web server's ENI.

   • Set Data Transfer Type to All Traffic.

   • To configure delivery, select Deliver to Log Service and Enable Log Analysis Report.

   • Retain the default values for other parameters.

2. Analyze the flow logs

   1. Query and analysis statement

      Filter source IPs that access port 80 of 10.0.0.1, and display the number of times each IP is allowed and rejected by the security group:

      dstaddr:10.0.0.1 AND dstport:80 | SELECT -- Filter logs where the destination IP address is 10.0.0.1 and the destination port is 80.
      srcaddr,
      SUM(CASE WHEN action = 'ACCEPT' THEN 1 ELSE 0 END) AS accept_count, -- Count 1 for each ACCEPT (allowed) action.
      SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) AS reject_count -- Count 1 for each REJECT (denied) action.
      FROM log
      GROUP BY srcaddr -- Group by source IP address.
      ORDER BY accept_count + reject_count DESC -- Sort the results in descending order based on the total number of allowed and denied actions.

   2. Preview

      

      As shown in the figure, the srcaddr column displays the source IPs that access port 80. The accept_count and reject_count columns show the flow log entries for each source IP allowed and rejected by the security group within the query time range.

      • Five source IPs accessed port 80: 120.26.XX.XX, 121.43.XX.XX, 154.212.XX.XX, 176.65.XX.XX, and 198.235.XX.XX.

      • All requests from 120.26.XX.XX were allowed, while all requests from the other public IP addresses were rejected.

Ttraffic between ECS instances

Item	ECS instances in a VPC	ECS instances in different VPCs
Diagram
Description	Assume that you have deployed three ECS instances in one VPC, and there is traffic between the ECS instances. Use the flow log feature to analyze the traffic rate and trends between the ECS instances.	You have two VPCs in different regions, and each VPC has multiple ECS instances. The two VPCs are connected using a VPC peering connection. The peering connection is billed on a pay-as-you-go basis for Cloud Data Transfer (CDT). You notice that the inter-region data transfer fee has significantly increased. Use flow logs to identify the ECS instances that generate high volumes of traffic and reduce traffic costs.
Flow log configuration	For Resource Instance, select the ENI associated with ECS1. For Analysis and Delivery, select Deliver to Log Service and Enable Log Analysis Report. Retain the default values for other parameters.	For Resource Instance, select VPC VPC1. In the Analysis and Delivery section, select Deliver to Log Service and Enable Log Analysis Report. Retain the default values for other parameters.
Query and statement	Query the traffic rate trends between ECS1 and other ECS instances: (srcaddr:10.0.0.1 AND dstaddr:10.0.0.*) OR (srcaddr:10.0.0.* AND dstaddr:10.0.0.1 ) | select --Filter traffic trends between ECS1 and other ECS instances. date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the UNIX timestamp to a readable time format. concat(srcaddr,'->', dstaddr) as src_to_dst, -- Concatenate IP session pairs in the "source_ip->destination_ip" format. sum(bytes*8/60) as bandwidth -- Convert bytes to bits and divide by the 1-minute capture window. group by time,srcaddr,dstaddr -- Group by time, source IP address, and destination IP address. order by time asc -- Sort by time in ascending order. limit 100 -- Display the first 100 results.	Query the session traffic rate trends between the two VPCs: (srcaddr:10.0.* AND dstaddr:172.16.*) OR (srcaddr:172.16.* AND dstaddr:10.0.*) | select --Filter sessions between the two VPCs. date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the UNIX timestamp to a readable time format. concat(srcaddr,'->', dstaddr) as src_to_dst, -- Concatenate IP session pairs in the "source_ip->destination_ip" format. sum(bytes*8/60) as bandwidth -- Convert bytes to bits and divide by the 1-minute capture window. group by time,srcaddr,dstaddr -- Group by time, source IP address, and destination IP address. order by time asc -- Sort by time in ascending order. limit 100 -- Display the first 100 results.
Preview	The traffic rate from 10.0.0.1 to 10.0.0.2 is the highest, at approximately 1.4 Mbps. The traffic rate from 10.0.0.1 to 10.0.0.3 is the second highest, at approximately 700 Kbps. Other traffic accounts for a small proportion.	As shown in the figure, the traffic from 10.0.0.1 to 172.16.0.1 is a traffic spike of approximately 6 Mbps.

Internet NAT gateway traffic

You have multiple ECS instances in a region. All instances are deployed in the same vSwitch and access the Internet through the SNAT feature of an Internet NAT gateway.

You notice that the volume of NAT traffic to the Internet has significantly increased, which causes slow server responses. Use flow logs to identify the ECS instances that generate a high proportion of the traffic.

1. Create a flow log

   • For Resource Instance, select vSwitch 2, the vSwitch for the Internet NAT gateway.

   • In the delivery configuration, select Deliver to Log Service and Enable Log Analysis Report.

   • Retain the default values for other parameters.

2. Filter specific traffic paths

   To filter traffic on specific paths, you must specify conditions in the query statement:

   Diagram	No.	Filter method
   	1	Filter traffic from an ECS instance to the NAT gateway: Set direction to in and srcaddr to the private IP address of the ECS instance.
   	2	Filter traffic from the NAT gateway to the Internet: Set direction to out and srcaddr to the private IP address of the NAT gateway.
   	3	Filter traffic from the Internet to the NAT gateway: Set direction to in and dstaddr to the private IP address of the NAT gateway.
   	4	Filter traffic from the NAT gateway to an ECS instance: Set direction to out and dstaddr to the private IP address of the ECS instance.

3. Analyze the flow logs

   1. Query and analysis statement

      On the path from an ECS instance to the NAT gateway, analyze traffic to a specific public IP address:

      direction: 'in' and srcaddr: 10.0.0.* and dstaddr: 120.26.XX.XX | select -- Filter logs of ECS instances that access a specific public IP address.
      date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, srcaddr,  -- Convert the UNIX timestamp to a readable time format.
      sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the 1-minute capture window.
      group by time,srcaddr -- Group by time and source IP address.
      order by time asc  -- Sort by time in ascending order.
      limit 100 -- Display the first 100 results.

      Other common statements

      • On the path from the NAT gateway to an ECS instance, filter inbound traffic from a specific public IP address to all ECS instances:

        direction: 'out' and dstaddr: 10.0.0.* and srcaddr: 120.26.XX.XX | select -- Filter logs of ECS instances that access a specific public IP address.
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,   -- Convert the UNIX timestamp to a readable time format.
        dstaddr,
        sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the 1-minute capture window.
        group by time,dstaddr -- Group by time and destination IP address.
        order by time asc  -- Sort by time in ascending order.
        limit 100 -- Display the first 100 results.

      • On the path from an ECS instance to the NAT gateway, filter outbound traffic from ECS instances to all public IP addresses:

        direction: 'in' and srcaddr: 10.0.0.*  | select -- Filter logs of ECS instances that access all public IP addresses.
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,  -- Convert the UNIX timestamp to a readable time format.
        concat(srcaddr,'->', dstaddr), -- Concatenate IP session pairs in the "source_ip->destination_ip" format.
        sum(bytes*8/60) as bandwidth  -- Convert bytes to bits and divide by the 1-minute capture window.
        group by time,srcaddr,dstaddr -- Group by time, source IP address, and destination IP address.
        order by time asc  -- Sort by time in ascending order.
        limit 100 -- Display the first 100 results.

   2. Preview

      

      As shown in the figure, on the path from the ECS instance to the NAT gateway, the traffic rate from 10.0.0.1 (ECS1) to the public IP address 120.26.XX.XX is the highest, at approximately 12 Kbps.

Traffic in Express Connect circuit

An enterprise uses two VPCs in an Alibaba Cloud region to deploy different services. It connects its on-premises data center to Alibaba Cloud using an Express Connect circuit and CEN.

The IT department plans to use flow logs to monitor and analyze how traffic from different services in the VPCs uses the Express Connect circuit. This analysis helps with network resource planning and performance improvements.

1. Create a flow log

   Create two flow logs that deliver data to the same Logstore. The key parameters for each flow log are as follows:

   • For Resource Instance, under VPC, select VPC1 and VPC2.

   • Set Sampling Path to Traffic Through Transit Routers.

   • In the delivery configuration, select Deliver to Log Service, choose the same Logstore for both flow logs, and select Enable Log Analysis Report.

   • Retain the default values for other parameters.

2. Analyze the flow logs

   1. Query and analysis statement

      Analyze the proportion of traffic that flows to the on-premises data center:

      action: ACCEPT and srcaddr: 192.168.* and dstaddr:10.1.* | 
      WITH 
          vpc1_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc1_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.20.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          ),
          vpc2_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc2_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.10.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          )
      SELECT 
          COALESCE(vpc1_traffic.minute, vpc2_traffic.minute) AS minute,
          (COALESCE(vpc1_traffic.total_vpc1_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc1_percentage, 
          (COALESCE(vpc2_traffic.total_vpc2_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc2_percentage
      FROM vpc1_traffic FULL OUTER JOIN vpc2_traffic ON vpc1_traffic.minute = vpc2_traffic.minute 
      ORDER BY minute

      Click to view an explanation of the SQL statement.

      • Filter conditions:

        • srcaddr: 192.168.* filters logs from source addresses that start with 192.168.*.

        • dstaddr: 10.1.* filters logs where the destination address starts with 10.1.*.

        • action: ACCEPT filters logs where the value of the action field is ACCEPT.

      • Main query

        • A FULL OUTER JOIN is used to combine the results of vpc1_traffic and vpc2_traffic based on the minute field.

        • Calculates the percentage of traffic for each VPC per minute:

          • vpc1_percentage represents the percentage of traffic from VPC1 in the total traffic.

          • vpc2_percentage represents the percentage of traffic from VPC2 in the total traffic.

        • The query results are sorted in ascending order by minute.

      • WITH subquery:

        The SQL statement contains two subqueries: vpc1_traffic and vpc2_traffic. The following describes the vpc1_traffic subquery:

        • The date_trunc function truncates the UNIX timestamp in the __time__ field to the minute, and the result is aliased as minute.

        • The SUM function is used to calculate the total traffic rate in bit/s over a one-minute period, and the result is named total_vpc1_traffic.

        • Filters traffic records with source addresses of 192.168.20.*, which is the CIDR block of VPC1.

        • Groups the results by minute.

   2. Preview

      

      During the period from 14:50 to 15:50, the proportion of traffic from VPC1 that flows to the on-premises data center is high.