Stress test an ALB instance

Application Load Balancer (ALB) has four performance metrics, each requiring a specific connection type and testing approach. Mismatched settings, tooling limitations, or client-side bottlenecks cause inaccurate results.

Test topology

Use the following test architecture:

Key metrics and connection types

ALB has four performance metrics. Use the recommended connection type for each.

Metric	Connection type	Notes
New connections	Short connections	Use a heartbeat service to minimize bandwidth impact. Watch for client port exhaustion.
Concurrent connections	Long connections	Keep connections alive with a heartbeat service. Watch for client port exhaustion at high concurrency.
Data throughput	Long connections	Set a short timeout (for example, 5 seconds). A long timeout inflates the average response time and obscures whether you reached the target load. A short timeout surfaces issues through the success rate, making the actual load easier to identify.
Requests per second	HTTP short, HTTP long, HTTPS short, or HTTPS long connections	Measures request processing capacity. Any of the four connection types works.

Stress testing tools

Do not use Apache ab for stress testing:

Apache ab exhibits staircase-pattern pauses of 3, 6, and 9 seconds under high concurrency.
Apache ab uses content length to determine whether a request succeeds. When ALB distributes traffic across multiple backend servers, each server may return a different content length, which causes inaccurate test results.

Disable health checks during stress tests

Disable health checks on the server group before running stress tests. Health check traffic adds extra requests to backend servers and skews measurements. If a backend server flaps between healthy and unhealthy states, ALB repeatedly adds and removes it from the rotation, which reduces measured performance.

Troubleshoot low performance

If stress test results are lower than expected, check for these common causes.

Client port exhaustion

ALB strips the TCP timestamp attribute by default, which prevents tw_reuse (TIME_WAIT connection reuse) from working in the Linux protocol stack. Connections in TIME_WAIT state accumulate and exhaust available client ports.

Solution:

Replace short connections with long connections.
Use RST packets to close connections by setting the SO_LINGER socket option.

Backend server accept queue full

When the backend server accept queue is full, the server cannot send SYN_ACK packets, causing client-side timeouts.

Solution: The default value of net.core.somaxconn is 128. Increase this value and then restart the application on the backend server:

sysctl -w net.core.somaxconn=1024

Application dependency bottleneck

Backend server load may appear normal, but dependent services such as databases can bottleneck and degrade overall performance.

Solution: Remove unnecessary applications from backend servers, or scale out dependent services.