Benchmark PolarSearch with OpenSearch Benchmark - PolarDB

Test tool

OpenSearch Benchmark (OSB) is an open-source benchmarking framework from the OpenSearch project, formerly Elasticsearch Rally. It provides standardized workloads for repeatable performance benchmarks on any Elasticsearch/OpenSearch-compatible search engine.

Project homepage: https://github.com/opensearch-project/opensearch-benchmark
Documentation: https://docs.opensearch.org/docs/latest/benchmark/

Test environment

Test tool: OpenSearch Benchmark 2.1.0
Setup: The ECS instance and the PolarSearch cluster must be in the same region, availability zone, and VPC network.
ECS instance:
- Instance type: ecs.c9i.8xlarge (32 cores, 128 GiB)
- Operating system: Ubuntu 22.04
- Python version: 3.10+
PolarSearch cluster:
- Node type: 8 cores, 32 GB
- Number of nodes: 2
- Parameter configuration: Default configuration with no parameter adjustments.

Products and versions

Product	PolarSearch version	Node type	Number of nodes
PolarSearch	3.0	8 cores, 32 GB	2
PolarSearch	1.0	8 cores, 32 GB	2

Workloads

Workload	Description	Test operations
HTTP logs	Web server access logs from the 1998 FIFA World Cup (~247 million records).	index-append data ingestion, term query, range time query, aggregation
NYC taxis	New York City taxi trip data from 2015.	index-append data ingestion, term query, range time query, geodistance query, aggregation
Geonames	GeoNames geographical database, allCountries export from April 2017 (~11.4 million records).	term and phrase (exact/full-text) queries, aggregation, decay_geo_gauss_function_score, painless_static script scoring, desc_sort_population sort query

Test scenarios

Indexing performance test

Batch-writes the full dataset and measures write throughput at different concurrency levels (bulk_indexing_clients).
Uses the append-no-conflicts-index-only procedure, which creates an index and ingests data without search operations. Each run starts with an empty index.
Concurrent indexing clients tested at 1, 2, 4, 8, 16, and 32.

Search performance test

After ingestion, use --include-tasks to run search tests on existing data without re-importing.
Concurrent search clients tested at 1, 2, 4, 8, 16, and 32, with target_throughput set to 0 (full-speed mode).

Install OpenSearch Benchmark

Note

OpenSearch Benchmark requires Python 3.8 or later.

Install OpenSearch Benchmark.
```
pip install opensearch-benchmark
```
Verify the installation.
```
opensearch-benchmark --version
```

Test procedure

Prerequisites

Obtain the endpoint, username, and password for your PolarSearch cluster and verify network connectivity:

curl -u <user>:<password> http://<endpoint>/_cluster/health?pretty

Response status descriptions:

"status": "green": The cluster is healthy.
"status": "yellow": All primary shards are allocated, but some replica shards are not (for example, a node is offline). The cluster can still process read and write requests, but its high availability is reduced. Use the following commands to investigate:
```
# Check shard allocation status
curl -u <user>:<password> -XGET "http://<endpoint>/_cat/shards?v"
```
```
# View details about unassigned shards
curl -u <user>:<password> -XGET "http://<endpoint>/_allocation/explain"
```
"status": "red": At least one primary shard is unassigned, making some data unavailable. Do not run performance tests when the cluster is in this state.

Run the performance test

Start the performance test.

opensearch-benchmark run \
    --workload="<workload>" \
    --client-options="basic_auth_user:<user>,basic_auth_password:<password>,verify_certs:false" \
    --target-hosts="<endpoint>" \
    --pipeline=benchmark-only \
    --results-file="path/to/result_file.md" \
    --kill-running-processes \
    --workload-params="number_of_replicas:<num_replicas>,number_of_shards:<num_shards>,bulk_indexing_clients:<num_indexing_clients>,search_clients:<num_search_clients>,target_throughput:0"
    
    
# Optional parameters
#   --test-procedure="<test_procedure_name>"
#   --include-tasks="<task_names>"

General command-line parameters:

Parameter	Description
--workload	The name of the test workload, such as `http_logs` or `nyc_taxis`.
--test-procedure	Test procedure name. Use `append-no-conflicts-index-only` for write-only tests, or `append-no-conflicts` for search tests. Combine with `--include-tasks` to run only the search part. Note Run `opensearch-benchmark info --workload=<workload_name>` to list available test procedures.
--include-tasks	Runs only the listed tasks, skipping index deletion, creation, and ingestion. Use this to search-test existing data.
--client-options	Authentication credentials for the cluster. Note `verify_certs:false` skips SSL certificate verification.
--target-hosts	The endpoint of the cluster under test.
--pipeline=benchmark-only	Runs the benchmark against an external PolarSearch cluster.
--results-file	Output path for test results (Markdown format).
--kill-running-processes	Terminates lingering OSB processes from previous runs.
--workload-params	Comma-separated key-value pairs that override workload template defaults.

Workload parameters:

Parameter	Description
number_of_replicas	The number of replica shards for the index.
number_of_shards	The number of primary shards for the index.
bulk_indexing_clients	The number of concurrent indexing clients.
search_clients	The number of concurrent search clients; applies during search tests.
target_throughput:0	Disables rate limiting, allowing the test to run at full speed to measure the cluster's maximum search throughput.

Test results

After each test, OSB prints a summary to the console and writes detailed results to the --results-file path.

Metric	Description
Mean throughput	Average throughput in ops/s (operations per second). With `target_throughput=0`, this value reflects the maximum processing capacity of the cluster at the given concurrency. Higher values indicate better performance.
p50 latency	50th percentile (median) latency. Reflects typical request latency.
p90 / p99 latency	90th and 99th percentile latencies, reflecting long-tail request latency. Lower values indicate more stable performance under high load.
Service time	Time from request to response, excluding queue wait. Reflects pure processing time. Also reported as p50, p90, and p99 percentiles.
Error rate	Percentage of failed requests. Valid results require 0%. A non-zero rate indicates errors under load (circuit breaking, timeouts), making that run's data unreliable.

Detailed results are available in PolarSearch 1.0 Performance Test Results and PolarSearch 3.0 Performance Test Results.