Analyze high-traffic ECS instances in an Internet NAT Gateway - NAT Gateway

A sudden spike in internet traffic can cause your Internet NAT Gateway to reach its capacity and bandwidth limits. This can slow down your services and disrupt business operations. You can use VPC flow logs to analyze traffic from ECS instances that use SNAT rules in an Internet NAT Gateway. By identifying instances with high traffic, you can optimize network bandwidth allocation and resolve network bottlenecks.

Overview

Internet NAT Gateway is an enterprise-grade service that provides SNAT and DNAT features. It allows you to perform detailed monitoring and management for instances that access the internet. You can also use an Internet NAT Gateway to view top traffic contributors, pinpoint ECS instances with sudden traffic spikes, and access a wide range of traffic monitoring metrics. For more information, see Internet NAT Gateway.

A VPC flow log can capture traffic from a specific elastic network interface (ENI), a specific VPC, or all ENIs in a vSwitch.

When you need to troubleshoot traffic from ECS instances that use an Internet NAT Gateway for SNAT or DNAT, you can create a VPC flow log. The flow log captures traffic from all ENIs in the vSwitch where the NAT gateway resides. By enabling and analyzing the VPC flow log, you can systematically investigate traffic from each ECS instance, effectively manage and optimize high-traffic instances, and ensure stable network performance.

Key features

After a VPC flow log is enabled, it collects and stores traffic log data. You can view and analyze the traffic logs for each path an ECS instance uses to access the internet. This provides a comprehensive view of each instance's network traffic.

Traffic between your internal network and the internet is routed through an Internet NAT Gateway. The ENI associated with the Internet NAT Gateway serves as the interface for traffic that enters and leaves the gateway. You can monitor the inbound and outbound traffic on this ENI to track traffic usage and bandwidth consumption for each path.

As shown in the following figure, the traffic path from an ECS instance to the internet through an Internet NAT Gateway is divided into four segments. The ENI of the NAT gateway serves as the dividing point.

You can use the Flow Log Center to view and analyze traffic in the IN direction and OUT direction, and to identify which ENIs are being monitored.

Segment log examples

Log on to the Log Service console to view traffic data and specific fields for these four segments. For more information about the fields recorded in flow logs, see Flow log records.

Segment	Log example
①	For segment ①, the traffic direction is inbound. The source and destination IP addresses are as follows: Source IP address: The private IP address of the ECS instance. Destination IP address: A specific public IP address.
②	For segment ②, the traffic direction is outbound. The source and destination IP addresses are as follows: Source IP address: The private IP address of the NAT gateway. Destination IP address: A specific public IP address.
③	For segment ③, the traffic direction is inbound. The source and destination IP addresses are as follows: Source IP address: A specific public IP address. Destination IP address: The private IP address of the NAT gateway.
④	For segment ④, the traffic direction is outbound. The source and destination IP addresses are as follows: Source IP address: A specific public IP address. Destination IP address: The private IP address of the ECS instance.

Scenarios

You can use flow logs to identify high-traffic ECS instances that use an Internet NAT Gateway in the following scenarios:

Optimize network performance: In high-concurrency and high-traffic business scenarios, you can use flow logs to analyze the outbound and inbound traffic of each ECS instance that uses the Internet NAT Gateway. This helps you identify high-traffic instances and pinpoint which instance and source IP address combinations cause bandwidth bottlenecks. You can then allocate bandwidth resources more effectively to prevent network congestion caused by overloaded instances.
Control and optimize costs: High-traffic instances can lead to significant bandwidth costs. By analyzing traffic logs, you can identify which instances and source IP addresses consistently generate high traffic. You can then optimize the network access paths for these instances to reduce unnecessary traffic and lower your costs.

Example scenario

A company runs multiple internal servers that use the SNAT feature of an Internet NAT Gateway to access the internet. Recently, the company noticed that it takes longer for the internal servers to access internet resources, which degrades the user experience. The company wants to identify the servers that have high traffic and take measures, such as optimizing and planning bandwidth resources, to resolve these performance issues.

This topic uses the scenario shown in the following figure as an example. A company has a VPC in the China (Hangzhou) region. The VPC contains three ECS instances in vSwitch1. These instances use the SNAT feature of an Internet NAT Gateway in vSwitch2 to access the internet. A sudden increase in the traffic from the internal servers to the internet slows down server responses and degrades user experience. The company needs to use VPC flow logs to identify which of the three ECS instances has high traffic. This information will guide resource re-planning to resolve network performance bottlenecks.

Prerequisites

You have created a VPC, vSwitch1, and vSwitch2 in the China (Hangzhou) region. For more information, see Create and manage a VPC.
You have created three ECS instances in vSwitch1: ECS01, ECS02, and ECS03. For more information, see Create an instance by using the wizard.

You have created an Internet NAT Gateway in vSwitch2 and created a vSwitch-level SNAT entry for the NAT gateway. The SNAT entry applies to vSwitch1. For more information, see Use the SNAT feature of an Internet NAT gateway to access the internet.

Parameters

Parameter	Value
VPC	VPC CIDR block: 172.16.0.0/12
vSwitch1 CIDR block	vSwitch CIDR block: 172.16.1.0/24
vSwitch2 CIDR block	vSwitch CIDR block: 172.16.3.0/24
Internet NAT Gateway	Private IP address of the Internet NAT Gateway: 172.16.3.128
elastic IP address	118.XX.XX.86
ECS instances	ECS01 instance: 172.16.1.44 ECS02 instance: 172.16.1.45 ECS03 instance: 172.16.1.46

Procedure

Step 1: Create a flow log

Before you create a flow log, ensure that you meet the prerequisites. For more information, see Prerequisites for using flow logs.

Log on to the VPC console.
In the left-side navigation pane, choose O&M and Monitoring > Flow Log.
In the top navigation bar, select the target region. In this example, China (Hangzhou) is selected.
On the Flow Log page, click Create a flow log.

In the Create a flow log dialog box, configure the parameters as described in the following table and then click OK.

Parameter	Description
Resource Type	Select the type of resource for which you want to capture traffic. In this example, vSwitch is selected.
Resource Instance	Select the resource instance for which you want to capture traffic. In this example, the instance ID of vSwitch1 is selected.
Data Transfer Type	Select the type of traffic to capture. For this example, select {value}.
Analysis and Delivery	Select Mode: Select Deliver to Log Service. Then, configure the Project and Logstore. Select Enable Log Analysis Report. This enables indexing for the selected Logstore and creates a dashboard to support SQL-based and visual analysis of the data.

Step 2: Simulate user traffic

Log on to the ECS01, ECS02, and ECS03 instances.
Run the following commands on each of the ECS01, ECS02, and ECS03 instances to install the wrk tool.
```
yum -y install git make gcc
git clone https://github.com/wg/wrk.git
yum install unzip
cd wrk
make
```
Run the following commands on the ECS01, ECS02, and ECS03 instances to simulate user traffic.

On ECS01, run the following command.
```
./wrk -c 1000 -d 60s -t 3  http://101.XX.XX.200:80/  # Replace 101.XX.XX.200 with a specific public IP address. 
```
On ECS02, run the following command.
```
./wrk -c 2000 -d 60s -t 3  http://101.XX.XX.200:80/  # Replace 101.XX.XX.200 with a specific public IP address.
```
On ECS03, run the following command.
```
./wrk -c 3000 -d 60s -t 3  http://101.XX.XX.200:80/  # Replace 101.XX.XX.200 with a specific public IP address.
```
The parameters in the commands are described as follows:
- -c: The number of concurrent connections to maintain for each thread.
- -d: The test duration. s specifies the unit in seconds.
- -t: The number of threads to use. Each thread simulates a concurrent user.

Step 3: View the flow log

Log on to the VPC console.
In the left-side navigation pane, choose O&M and Monitoring > Flow Log.
In the top navigation bar, select the target region. In this example, China (Hangzhou) is selected.
On the Flow Log page, find the flow log that you created. In the Simple Log Service column, click the name of the Logstore to go to the Log Service console and view the traffic information.

Follow the steps in the table below to view the traffic of the ECS instances that access the internet through the SNAT entry of the Internet NAT Gateway.

The flow log shows that the ECS03 instance, with the private IP address 172.16.1.46, has the highest traffic.

Step	Description
①	The following SQL query aggregates and sorts the collected VPC flow logs to create a chart showing the ECS instances that generate high traffic to a specific public IP address. `dstaddr: "101.XX.XX.200" and action: ACCEPT and srcaddr: 172.16.1.* \| select date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, srcaddr,sum(bytes8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) as bandwidth group by time,srcaddr order by time asc limit 1000` This SQL query defines three fields: `time`, `bandwidth` (in bps), and `srcaddr` (source address). The query results are grouped by `time` and `srcaddr`, sorted by `time` in ascending order, and limited to 1,000 log entries. For more information about the fields, see VPC flow log fields. The parameters are described as follows: `dstaddr`: The destination address, which is the public IP address. In this example, it is `101.XX.XX.200`. `srcaddr`: The source address, which is the private IP address of an ECS instance. In this example, it is `172.16.1.`. For other fields, use the values in the example. Note To filter for inbound traffic from a specific public IP address to the ECS instances, use the following SQL query: `srcaddr: "101.XX.XX.200" and action: ACCEPT and dstaddr: 172.16.1.* \| select date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, dstaddr,sum(bytes8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) as bandwidth group by time,dstaddr order by time asc limit 1000` The parameters are described as follows: `srcaddr`: The source address, which is the public IP address. In this example, it is `101.XX.XX.200`. `dstaddr`: The destination address, which is the private IP address of an ECS instance. In this example, it is `172.16.1.`. When you generate the chart, set Aggregate Column to `dstaddr`. To filter for outbound traffic from the ECS instances to all public IP addresses, use the following SQL query: `srcaddr: 172.16.1.* and action: ACCEPT \| select date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as time, srcaddr,sum(bytes8/(case WHEN "end"-start=0 THEN 1 else "end"-start end)) as bandwidth from log where ip_to_domain(dstaddr)!='intranet' group by time,srcaddr order by time asc limit 1000` The parameters are described as follows: `srcaddr`: The source address, which is a private IP address. In this example, it is `172.16.1.`. `dstaddr`: The destination address, which is a public IP address. When you generate the chart, set Aggregate Column to `srcaddr`.
②	Select a time range for the query. In this example, Last 5 Minutes is selected.
③	Click the General Configurations tab, and then click the icon to select the line chart format.
④	In the Search & Analysis Settings section, configure the following parameters: Axis X Field: Set this to time. Axis Y Field: Set this to bandwidth. Aggregate Column: Set this to srcaddr. In the Standard Configurations section, set Format to bits/s (SI). Keep the default values for the other parameters.
⑤	Click Add to New Dashboard and configure the following parameters in the dialog box: Operation: In this example, Create Dashboard is selected. Layout Mode: In this example, Grid Layout is selected. Dashboard Name: Enter a name for the dashboard. In this example, Outbound ECS traffic via NAT Gateway is entered. The Dashboard shows the flow log information.
⑥	Click Search & Analyze to view the outbound traffic from each ECS instance and identify the high-traffic instances.

References

For more information about the fields captured by VPC flow logs, see Flow log records.
If an error occurs when you query a flow log, see Common errors in log queries and analysis to troubleshoot the issue.
For more information about how to query and analyze logs, see Quick start for log query and analysis.