Elizabeth
Engineer
Engineer
  • UID625
  • Fans4
  • Follows1
  • Posts68
Reads:2756Replies:0

Benchmark testing of OSS with ab and wrk tools

Created#
More Posted time:Oct 27, 2016 16:58 PM
Background
With the increasing number of OSS users, many want to know what performance OSS can offer.
Here the performance mainly refers to the query per second (QPS), and the latency of every processing request.
Testing the performance of OSS is quite a broad topic.
From the users' perspective, OSS performance is related to the stress type (synchronous or asynchronous), the size of the requested object as well as the request method (read or write).
From the OSS service end perspective, the performance provided is related to the server model (disks, NICs, memories, and CPUs), the number of servers as well as the network and load of the entire cluster.
Scope of usage
This article talks about:
How to use the ab and wrk tools
• In a single-server environment
• Send requests to OSS endpoints
• Calculate the request QPS and latency on the client side.
Tools
The ab tool
The full name of ab is apache benchmark, an official tool launched by Apache.
The tool is used to test the performance of Apache servers. It views the service capacity provided by the installed Apache servers and the number of requests processed per second.
How to get and install ab
http://httpd.apache.org/docs/2.4/install.html
http://httpd.apache.org/docs/2.4/programs/ab.html
The ab tool will be compiled together when Apache servers are compiled. That compilation procedure won’t be covered here.
Usage
Because OSS buckets have permissions, and ab does not support OSS signatures, the buckets should be set to the public-read-write permission for testing.
Suppose we simulate a scenario with 10 concurrent requests and the requested object is 100KB in size.
Frequently-used configuration items during ab execution:
-c Number of concurrent requests
The total number of requests sent once. By default, one request is sent once.
-k Maintain keep-alive
Enable keep-alive, and multiple requests are allowed in an HTTP session. The configuration item is disabled by default.
-n Number of requests
The number of requests required throughout the benchmark test.
The default is one. The performance parameters in default mode are not representative.
-T The maximum time
The longest time of the benchmark test. No restrictions are imposed by default.
-u Upload files
File containing data to PUT. Remember to also set -T.-T content-type
-T To set the Content-Type for the files uploaded.
For example: application/x-www-form-urlencoded. Default value is text/plain.
Examples
• Test the performance of OSS in high-concurrency reads/writes to small files
• Precondition
A bucket with the public-read-write permission has been created. Its name can be public. The ab tool (open-source) has been downloaded under the Linux running environment. The OSS provides the service endpoint, for example the oss-cn-hangzhou-test.aliyuncs.com. Prepare a file of 5KB, for example 5KB.txt.
• Testing process
1. Simulate writes to small file (5KB) with a high concurrency (50 threads). Run ./ab -c 50 -t 300 -T ‘text/plain’ -u 5KB.txt for 5 minutes.
2. Simulate reads of small file (5KB) with a high concurrency (50 threads). Run ./ab -c 50 -t 300 for 5 minutes.
• Expected results
The test is ended normally. Failed requests totaled 0, and the requests per second value indicates the processing capacity for the client per second. But this does not represent the processing capacity of the OSS service end.
Notes
• Observe the CPUs, memory and networks of the server where the ab testing tool is located and the tested front-end server to make sure they do not exceed 75% of the maximum values allowed.
• The test may fail because of insufficient ports.
At this time, you need to adjust the kernel parameters to support port reusage.
For example: on the Linux platform
1 sudo vim /etc/sysctl.conf
2. Add the following content
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
kernel.printk = 7 4 1 7
3. Run the sudo sysctl –p command to make it effective.
Result analysis
$./ab -c 50 -t 60 -n 300000 -k http://oss-cn-hangzhou-test.aliyuncs.com/public/5KB.txt
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking oss-cn-hangzhou-test.aliyuncs.com (be patient)
Completed 30000 requests
Completed 60000 requests
Completed 90000 requests
Completed 120000 requests
Completed 150000 requests
Completed 180000 requests
Completed 210000 requests
Completed 240000 requests
Finished 250137 requests


Server Software:        AliyunOSS
Server Hostname:        oss-cn-hangzhou-test.aliyuncs.com
Server Port:            80

Document Path:          /public/5KB.txt
Document Length:        5120 bytes

Concurrency Level:      50            
Time taken for tests:   60.000 seconds
Complete requests:      250137        
Failed requests:        0
Write errors:           0
Keep-Alive requests:    248492        
Total transferred:      1382504896 bytes
HTML transferred:       1280703929 bytes
Requests per second:    4168.94 [#/sec] (mean)  
Time per request:       11.993 [ms] (mean)      
Time per request:       0.240 [ms] (mean, across all concurrent requests)
Transfer rate:          22501.67 [Kbytes/sec] received

Connection Times (ms)    
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     1   12   7.6     12      87
Waiting:        1   12   7.6     12      87
Total:          1   12   7.6     12      87

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     15
  75%     16
  80%     17
  90%     20
  95%     23
  98%     28
  99%     37
 100%     87 (longest request)


From the testing results, we can see that
• In the case of 50 concurrent requests run for 60 seconds, and 4,168 times/requests could be handled per second (that is to say, the QPS shown on the client side was 4,168 at such a stress level).
• The latency of each processing request was around 12ms.
• Since the keep-alive was enabled, the connection basically took no time.
• 99% of requests were handled within 37ms, and the longest request lasted for 87ms.
wrk
The wrk tool is for HTTP benchmark testing. It can generate significant stress.
How to get and install wrk
https://github.com/wg/wrk
Usage
The wrk tool can be used in combination with lua scripts for PUT operations.
• Precondition > A bucket with the public-read-write permission has been created. Its name can be public. The wrk tool has been downloaded and installed under the Linux running environment. The OSS provides the service endpoint, for example the oss-cn-hangzhou-test.aliyuncs.com. Prepare a file of 5KB, for example 5KB.txt.
Upload
Here, the lua script is used for uploading operations. The content of the Lua script put.lua is as follows:
counter = 0
    request = function()
       mypath = "5KB.txt";
       local file = io.open(mypath, "r");
       assert(file);
       local body = file:read("*a");      -- Read all content
       file:close();
       wrk.method = "PUT"
       wrk.body = body
       path = "/public/test-" .. mypath .. "-" .. counter
       wrk.headers["X-Counter"] = counter
       counter = counter + 1
       return wrk.format(nil, path)
    end
    done = function(summary, latency, requests)
       io.write("------------------------------\n")
       for _, p in pairs({ 50, 60, 90, 95, 99, 99.999 }) do
          n = latency:percentile(p)
          io.write(string.format("%g%%, %d ms\n", p, n/1000.0))
       end
    end


Execute the command:
$./wrk -c 50 -d 60 -t 5 -s put.lua http://oss-cn-hangzhou-test.aliyuncs.com
It indicates to initiate a PUT request to the endpoint, and the requested content is specified in put.lua to own 5 threads, with 50 connections enabled and set to run for 60 seconds.


Test results:
Running input
-c 50 -d 60 -t 5 -s put.lua http://oss-cn-hangzhou-test.aliyuncs.com
Running 1m test @ http://oss-cn-hangzhou-test.aliyuncs.com, test input http://oss-cn-hangzhou-test.aliyuncs.com
  5 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    16.23ms    9.49ms 159.48ms   96.45%
    Req/Sec   635.38     98.38     0.91k    72.63%
  189072 requests in 1.00m, 48.73MB read
Requests/sec:   3151.10
Transfer/sec:    831.58KB
------------------------------
50%, 14 ms
60%, 15 ms
90%, 20 ms
95%, 23 ms
99%, 64 ms
99.999%, 159 ms


Result analysis:
From the testing results, we can see that
• With 5 concurrent requests, when 50 connections were enabled and the requests were run for 60 seconds, 3,151 times/requests could be processed on average per second (that is to say, under this stress, the QPS shown on the client was 3,151).
• The average latency of every request processing was around 16ms.
• 99% of requests were completed within 64ms, with the longest request lasting for 159ms.
Download
Execute the command:
$./wrk -c 50 -d 60 -t 5 http://oss-cn-hangzhou-test.aliyuncs.com/public/5KB.txt
It indicates to initiate a GET request to the endpoint, with 5 threads and 50 connections enabled. The request is run for 60 seconds.
Note: The 5KB.txt should exist here.


Test results:
Running input
-c 50 -d 60 -t 5 http://oss-cn-hangzhou-test.aliyuncs.com/public/5KB.txt
Running 1m test @ http://oss-cn-hangzhou-test.aliyuncs.com/public/5KB.txt, test input http://oss-cn-hangzhou-test.aliyuncs.com/public/5KB.txt
  5 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    12.72ms    5.14ms  62.68ms   80.14%
    Req/Sec   814.86    145.65     1.36k    69.43%
  241990 requests in 1.00m, 1.25GB read
Requests/sec:   4033.14
Transfer/sec:     21.26MB


Result analysis:
From the testing results, we can see that
• With 5 concurrent requests, when 50 connections were enabled and the requests were run for 60 seconds, 4033 times/requests could be processed on average per second (that is to say, under this stress, the QPS shown on the client was 4033).
• The latency of each processing request was around 12ms.
Summary
The above shows the QPS and latency shown on the client side as measured by the open-source benchmark tool.
But the performance shown on the client side is subject to the influences of various factors, such as the request methods, the local resources (CPU, memory and network), the OSS network conditions and OSS load.
You need to check whether the performance bottleneck is from OSS or from the local machine based on the actual conditions.

Latest likes:

binarybinary
Guest