Virtual Private Cloud:VPC flow log

Console

API

Before creating a flow log, ensure you have enabled the flow log feature and created a project and Logstore in Log Service (SLS):

• Call OpenFlowLogService to enable the flow log feature.

• Call CreateProject to create a project, and call CreateLogStore to create a Logstore.

After meeting these prerequisites, you can perform the following operations:

• Call CreateFlowLog to create a flow log, and optionally call CreateIndex to create an index.

• Call ModifyFlowLogAttribute to modify flow log attributes. You can modify the flow log name, description, aggregation interval, and custom subscription.

• Call DeactiveFlowLog to stop a flow log.

• Call ActiveFlowLog to start a flow log.

• Call DeleteFlowLog to delete a flow log.

Terraform

Resources: alicloud_log_project, alicloud_log_store, alicloud_vpc_flow_log

# Specify the region where you want to create the flow log.
provider "alicloud" {
  region = "cn-hangzhou"
}

# Specify the description of the project and the names of the Logstore and flow log.
variable "name" {
  default = "vpc-flowlog-example"
}

# Generate a random string to use in the project name.
resource "random_uuid" "example" {
}

# Create a Log Service project.
resource "alicloud_log_project" "example" {
  project_name = substr("tf-example-${replace(random_uuid.example.result, "-", "")}", 0, 16)
  description  = var.name
}

# Create a Log Service Logstore.
resource "alicloud_log_store" "example" {
  project_name          = alicloud_log_project.example.project_name
  logstore_name         = var.name
  shard_count           = 3
  auto_split            = true
  max_split_shard_count = 60
  append_meta           = true
}

# Create a VPC flow log.
resource "alicloud_vpc_flow_log" "example" {
  flow_log_name        = var.name
  log_store_name       = alicloud_log_store.example.logstore_name
  description          = var.name
  traffic_path         = ["all"] # Capture traffic from all scenarios.
  project_name         = alicloud_log_project.example.project_name
  resource_type        = "VPC" # Resource type.
  resource_id          = "vpc-bp1ekmgzch0bo3hxXXXXXX" # VPC ID.
  aggregation_interval = "1" # Aggregation interval in minutes.
  traffic_type         = "All" # Capture all allowed and rejected traffic.
}

Console

API

Call GetLogsV2 to query and analyze flow logs.

Source IPs for public access

In this scenario, you have a web server that is accessible from the Internet and listens on port 80. You have also configured security group rules to control access from specific source IPs.

By creating a flow log, you can identify the source IPs that access port 80 and determine how many of their requests your security group rules allow or reject.

1. Create a flow log

   • For Resource Instance, select the ENI of the web server.

   • For Data Transfer Type, select All Traffic.

   • For delivery settings, select Deliver to Log Service and enable the Enable Log Analysis Report.

   • Keep the default values for other settings.

2. Analyze the flow logs

   1. Query and analysis statement

      Filter for traffic to port 80 on the server at 10.0.0.1, and display the number of times each source IP was allowed or rejected by the security group:

      dstaddr:10.0.0.1 AND dstport:80 | SELECT -- Filter for logs where the destination IP is 10.0.0.1 and the destination port is 80
      srcaddr,
      SUM(CASE WHEN action = 'ACCEPT' THEN 1 ELSE 0 END) AS accept_count, -- Count 1 for each 'ACCEPT' (allowed) action
      SUM(CASE WHEN action = 'REJECT' THEN 1 ELSE 0 END) AS reject_count -- Count 1 for each 'REJECT' (rejected) action
      FROM log
      GROUP BY srcaddr -- Group the results by source IP address
      ORDER BY accept_count + reject_count DESC -- Sort results in descending order by the total number of allowed and rejected connections

   2. Result preview

      The srcaddr column lists the source IPs that accessed port 80. The accept_count and reject_count columns show how many times the security group allowed or rejected connections from each source IP during the specified time range:

      • Five source IPs accessed port 80: 120.26.XX.XX, 121.43.XX.XX, 154.212.XX.XX, 176.65.XX.XX, and 198.235.XX.XX.

      • All requests from 120.26.XX.XX were allowed, while all requests from the other public IP addresses were rejected.

Traffic between ECS instances

Parameter	Intra-VPC traffic	Inter-VPC traffic
Diagram
Description	Assume you have deployed three ECS instances in one VPC, and the instances communicate with each other. In this case, you can use flow logs to analyze the traffic rate and trends between the ECS instances.	A peering connection links two VPCs in different regions. You are billed for the cross-region traffic using the pay-as-you-go Cloud Data Transfer (CDT) model. You notice a recent, unexpected increase in your cross-region data transfer costs. In this case, use flow logs to identify which ECS instances generate the most traffic and help you optimize costs.
Flow log configuration	For Resource Instance, select the ENI of ECS1. For Analysis and Delivery, select Deliver to Log Service and enable the Enable Log Analysis Report. Keep the default values for other settings.	For Resource Instance, select VPC VPC1. For Analysis and Delivery, select Deliver to Log Service and enable the Enable Log Analysis Report. Keep the default values for other settings.
Query and analysis statement	Query the traffic rate trends between ECS1 and other ECS instances: (srcaddr:10.0.0.1 AND dstaddr:10.0.0.*) OR (srcaddr:10.0.0.* AND dstaddr:10.0.0.1 ) | select -- Filter for traffic trends between ECS1 and other ECS instances date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the Unix timestamp to a readable time format concat(srcaddr,'->', dstaddr) as src_to_dst, -- Create a session pair in the format 'source_ip->destination_ip' sum(bytes*8/60) as bandwidth -- Calculate bandwidth in bit/s by converting bytes to bits and dividing by the 60-second capture window group by time,srcaddr,dstaddr -- Group results by time, source IP, and destination IP order by time asc -- Sort by time in ascending order limit 100 -- Return the first 100 results	Query the session traffic rate trends between the two VPCs: (srcaddr:10.0.* AND dstaddr:172.16.*) OR (srcaddr:172.16.* AND dstaddr:10.0.*) | select -- Filter for sessions between the two VPCs date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, -- Convert the Unix timestamp to a readable time format concat(srcaddr,'->', dstaddr) as src_to_dst, -- Create a session pair in the format 'source_ip->destination_ip' sum(bytes*8/60) as bandwidth -- Calculate bandwidth in bit/s by converting bytes to bits and dividing by the 60-second capture window group by time,srcaddr,dstaddr -- Group results by time, source IP, and destination IP order by time asc -- Sort by time in ascending order limit 100 -- Return the first 100 results
Result preview	On the query and analysis results page, select the Graph tab and set the chart type to Area Chart. Set the x-axis field to time, the y-axis field to bandwidth, and the aggregation column to src_to_dst. Set the format unit to bps,Kbps,Mbps(bit). The traffic rate from 10.0.0.1 to 10.0.0.2 is the highest at approximately 1.4 Mbps. The rate from 10.0.0.1 to 10.0.0.3 is the second highest at approximately 700 Kbps. The remaining traffic is minimal.	The visualization configuration is the same as the intra-VPC scenario. Select the Area Chart type and set the aggregation column to src_to_dst. Traffic from 10.0.0.1 to 172.16.0.1 spikes to approximately 6 Mbps.

Internet NAT Gateway traffic

In this scenario, you have multiple ECS instances deployed in the same region and vSwitch. These instances use the SNAT feature of an Internet NAT Gateway to access the Internet.

You notice a recent surge in traffic from the Internet NAT Gateway to the Internet, which causes slow server responses. You can use flow logs to identify the ECS instances responsible for most of this traffic.

1. Create a flow log

   • For Resource Instance, select vSwitch 2, the vSwitch where the Internet NAT Gateway is located.

   • For delivery settings, select Deliver to Log Service and enable the Enable Log Analysis Report.

   • Keep the default values for other settings.

2. Filter specific traffic paths

   To filter traffic on specific paths in this scenario, you must specify different conditions in your query statement:

   Diagram	Number	Filter method
   	①	Filter traffic from an ECS instance to the NAT gateway: The direction is in, and the srcaddr is the private IP address of the ECS instance.
   	②	Filter traffic from the NAT gateway to the Internet: The direction is out, and the srcaddr is the private IP address of the NAT gateway.
   	③	Filter traffic from the Internet to the NAT gateway: The direction is in, and the dstaddr is the private IP address of the NAT gateway.
   	④	Filter traffic from the NAT gateway to an ECS instance: The direction is out, and the dstaddr is the private IP address of the ECS instance.

3. Analyze the flow logs

   1. Query and analysis statement

      On the path from an ECS instance to the NAT gateway, analyze traffic to a specific public IP address:

      direction: 'in' and srcaddr: 10.0.0.* and dstaddr: 120.26.XX.XX | select -- Filter for logs of ECS instances accessing a specific public IP address
      date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, srcaddr,  -- Convert the Unix timestamp to a readable time format
      sum(bytes*8/60) as bandwidth  -- Calculate bandwidth in bit/s by converting bytes to bits and dividing by the 60-second capture window
      group by time,srcaddr -- Group results by time and source IP
      order by time asc  -- Sort by time in ascending order
      limit 100 -- Return the first 100 results

      Other common queries

      • On the path from the NAT gateway to an ECS instance, filter inbound traffic from a specific public IP address to all ECS instances:

        direction: 'out' and dstaddr: 10.0.0.* and srcaddr: 120.26.XX.XX | select -- Filter for logs of ECS instances accessing a specific public IP address
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,   -- Convert the Unix timestamp to a readable time format
        dstaddr,
        sum(bytes*8/60) as bandwidth  -- Calculate bandwidth in bit/s by converting bytes to bits and dividing by the 60-second capture window
        group by time,dstaddr -- Group results by time and destination IP
        order by time asc  -- Sort by time in ascending order
        limit 100 -- Return the first 100 results

      • On the path from an ECS instance to the NAT gateway, filter outbound traffic from the ECS instance to all public IP addresses:

        direction: 'in' and srcaddr: 10.0.0.*  | select -- Filter for logs of ECS instances accessing all public IP addresses
        date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time,  -- Convert the Unix timestamp to a readable time format
        concat(srcaddr,'->', dstaddr), -- Create a session pair in the format 'source_ip->destination_ip'
        sum(bytes*8/60) as bandwidth  -- Calculate bandwidth in bit/s by converting bytes to bits and dividing by the 60-second capture window
        group by time,srcaddr,dstaddr -- Group results by time, source IP, and destination IP
        order by time asc  -- Sort by time in ascending order
        limit 100 -- Return the first 100 results

   2. Result preview

      The visualization configuration is similar to the ECS traffic scenario. Select the Area Chart type and set the aggregation column to srcaddr.

      On the path from ECS instances to the NAT gateway, the traffic rate from 10.0.0.1 (ECS1) to the public IP address 120.26.XX.XX is the highest, at approximately 12 Kbps.

Express Connect traffic distribution

In this scenario, a company uses two VPCs in an Alibaba Cloud region to deploy different services. The company connects its on-premises data center to Alibaba Cloud using an Express Connect circuit and CEN.

The IT department wants to use flow logs to monitor and analyze how traffic from different services in the VPCs uses the Express Connect circuit's resources. This analysis helps with network resource planning and performance improvements.

1. Create flow logs

   Create two flow logs that deliver data to the same Logstore. Configure the key parameters for each flow log as follows:

   • For Resource Instance, select VPC and then select VPC1 and VPC2 respectively.

   • For Sampling Path, select Transit Router.

   • For delivery settings, select Deliver to Log Service, choose the same Logstore for both flow logs, and enable the Enable Log Analysis Report.

   • Keep the default values for other settings.

2. Analyze the flow logs

   1. Query and analysis statement

      Analyze the percentage of traffic from each VPC to the on-premises data center:

      action: ACCEPT and srcaddr: 192.168.* and dstaddr:10.1.* | 
      WITH 
          vpc1_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc1_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.20.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          ),
          vpc2_traffic AS (
              SELECT 
                  date_trunc('minute',__time__) AS minute,
                  SUM(bytes*8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) AS total_vpc2_traffic
              FROM 
                  log
              WHERE 
                  srcaddr LIKE '192.168.10.%'
              GROUP BY 
                  date_trunc('minute',__time__)
          )
      SELECT 
          COALESCE(vpc1_traffic.minute, vpc2_traffic.minute) AS minute,
          (COALESCE(vpc1_traffic.total_vpc1_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc1_percentage, 
          (COALESCE(vpc2_traffic.total_vpc2_traffic, 0) * 100/ NULLIF((COALESCE(vpc1_traffic.total_vpc1_traffic, 0) + COALESCE(vpc2_traffic.total_vpc2_traffic, 0)), 0)) AS vpc2_percentage
      FROM vpc1_traffic FULL OUTER JOIN vpc2_traffic ON vpc1_traffic.minute = vpc2_traffic.minute 
      ORDER BY minute

      The following provides a detailed explanation of the SQL statement.

      • Filter conditions:

        • srcaddr: 192.168.* filters for logs where the source address starts with 192.168..

        • dstaddr: 10.1.* filters for logs where the destination address starts with 10.1..

        • action: ACCEPT. Filters logs where the action field is ACCEPT.

      • Main query

        • Uses FULL OUTER JOIN to combine the results from vpc1_traffic and vpc2_traffic based on the minute field.

        • Calculates the percentage of traffic from each VPC for every minute:

          • vpc1_percentage represents the percentage of total traffic that originates from VPC1.

          • vpc2_percentage represents the percentage of total traffic that originates from VPC2.

        • The query results are sorted in ascending order by minute.

      • WITH subqueries:

        The SQL statement contains two subqueries, vpc1_traffic and vpc2_traffic. The following is an explanation of the vpc1_traffic subquery:

        • The date_trunc function truncates the Unix timestamp (the __time__ field) to the nearest minute, aliased as minute.

        • The SUM function calculates the total traffic rate in bit/s for each minute, aliased as total_vpc1_traffic.

        • Filters traffic records where the source address is in the 192.168.20.* CIDR block (VPC1).

        • Groups the results by minute.

   2. Result preview

      The visualization configuration is similar to the ECS traffic scenario. Select the Area Chart type to display the traffic percentage trends from different VPCs over time.

      During the period from 14:50 to 15:50, VPC1 generated a higher percentage of the traffic to the on-premises data center.