The Layer-4 Server Load Balancer (SLB) uses open source software Linux Virtual Server (LVS) and Keepalived to achieve load balancing. The Layer-7 SLB uses Tengine to achieve load balancing. For the Layer-4 SLB, requests directly reach backend servers after going through LVS. For the Layer-7 SLB, requests reach backend servers after going through LVS and then through Tengine. The Layer-7 SLB has one more procedure than the Layer-4 SLB when forwarding incoming requests. Due to this additional procedure, the performance of the Layer-7 SLB is less efficient to that of the Layer-4 SLB.
When performing stress tests on a Layer-7 SLB instance, you may find that the performance is much lower than expected. Also, you may experience the performance of the SLB instance with two backend ECS instances is slower to one single backend ECS instance of the same specification. The following are additional possible causes:
Cause: Insufficient client ports.
Lack of client ports may lead to a connection failure especially during stress testing. The SLB erases the timestamp attribute of TCP connection by default, and the tw_reuse feature of Linux protocol stack (reuse time_wait connection) cannot take effect. As a result, the time_wait connections pile up, leading to insufficient client ports.
Resolution: Use persistent connection instead of short-lived connection on the client. Use the RST message, instead of sending a FIN packet, to disconnect (set the SO_LINGER attribute for the socket).
Cause: The accept queue of the backend server is full.
The accept queue of the backend server is full, so that the backend server does not return syn_ack packets and the client times out.
Resolution: The default value of net.core.somaxconn is 128. Run
sysctl -w net.core.somaxconn=1024to change its value, and restart the application on the backend server.
Cause: Too many connections to the backend server.
Persistent connections change to short-lived connections after going through Tengine when using the Layer-7 SLB due to the architecture design. Therefore, numerous connections are sent to the backend server, resulting in poor performance during stress test.
Cause: Poor application performance.
After SLB forwards requests to the backend server, the load of the backend server is normal, but the performance is not as good as expected because the applications deployed on the backend server depend on other applications such as databases. If the database reaches its performance bottleneck, this may cause poor performance.
Cause: Abnormal health check status of backend servers.
It is easy to ignore the health check status of backend servers when doing stress testing. Health check failure of a backend server or frequent change of the health check status (frequent changes from healthy to unhealthy or from unhealthy to healthy) also results in poor performance.
Note the following during the stress testing:
Use short-lived connections to test the forwarding capability of SLB.
In addition to testing the session persistence and balancing test, the main goal of stress testing is to test the forwarding capability of SLB. We recommend that you use short-lived connections to test the processing capabilities of SLB and backend servers. Note the insufficient client ports problem when using the short-lived connections to do the stress testing.
Use persistent connections to test the throughput of SLB. Persistent connection is used to test the bandwidth limit or special services.
We recommend that you set a smaller timeout value (5 seconds) for the stress test tool. If the timeout value is too large, the test result shows an increased average RT, making it difficult to determine whether the throughput limit of SLB has been reached. When the timeout value is small, the test result shows the success rate, making it easier to quickly determine the throughput limit of SLB.
Use static web pages for stress testing to prevent the application logic cost.
Use the following listener configurations for stress testing:
Do not enable session persistence. This may cause the stress to be concentrated on a particular backend server.
Disable health check of the listeners to reduce the health check requests to the backend servers.
Perform the stress testing on multiple (more than five) clients with dispersed source IP addresses to simulate the actual online situation.
Do not use Apache ab as the stress testing tool.
In high-concurrency scenarios, Apache ab may experience tiered interruptions at 3s, 6s, and 9s. Apache ab determines whether a request is successful based on the content length. When multiple backend servers are added to SLB, the returned content length will be inconsistent thereby the test result is adversely affected.