Performance Testing:Technical guide for performance testing

Choose an environment

System environments are classified into production, test, and other environments. Performance tests typically run in either a production environment or a dedicated test environment. Each has trade-offs:

Production environment: Run stress tests during off-peak hours to avoid affecting live services. Alibaba Cloud full-link stress testing provides a more efficient approach for production testing. After testing, clean up test data or filter it from Business Intelligence (BI) reports.

Test environment: Build at a fraction of production scale -- typically half, quarter, or one-eighth. Common strategies include deploying separate test clusters for specific applications within production, or sharing databases between environments. Import masked data from production (typically the last 6 to 12 months) to maintain data relevance.

Test environment requirements

Match the test environment to production as closely as possible:

Investigate the test environment

Before testing, investigate each layer of the test environment to establish a baseline for monitoring and bottleneck analysis:

Metric categories

Track metrics across four categories:

Define metric thresholds

Define thresholds for each metric before testing. Without clear thresholds, test results lack actionable meaning -- different stakeholders (development, operations, business) have different expectations and care about different metrics.

Business metric benchmarks:

Resource metric guidelines:

• Limit CPU utilization to 75% or below

• Avoid swap partition usage entirely

• Ideally, when a system reaches its capacity limit, a resource (CPU, memory, I/O) should be the bottleneck rather than an application-level issue. Adding resources to a resource-bound system scales its capacity; adding resources to an application-bound system does not

Role of business models

Each business operation (login, search, checkout, payment) consumes different amounts of system resources. The mix of business types and their relative proportions determines overall system capacity. If the test traffic mix does not match production, the results have no practical value.

For example, in e-commerce, different promotion types and product categories shift the ratio of read-heavy versus write-heavy operations. Accurate traffic modeling in PTS captures these patterns and reveals the true system bottleneck.

Select business types

Choose high-volume, high-risk, and high-growth business operations as representative test cases.

For systems already in production:

• Collect business types and volumes at different peak periods. If the mix varies significantly across time periods, define multiple business models

• Identify time windows with high resource consumption or anomalies during peak hours, and investigate their root causes

• Review past production incidents. If an incident was caused by a business operation that was excluded from previous tests, add it to future test models

For systems not yet in production:

• Determine business types and proportions through stakeholder interviews and requirements analysis

• Assess whether specific business operations could spike during promotions or events

• After initial test runs, review resource consumption per operation. If a low-proportion operation consumes disproportionate resources, adjust its weight in the model

Basic data volume

Basic data volume refers to existing data in the database -- historical records, reference data, and accumulated business data. The volume of existing data directly affects query performance: a search across thousands of records behaves very differently from one across millions.

Requirements:

• Match the test environment's data volume to the same order of magnitude as production

• Account for data growth over the next three years. If data grows rapidly, load additional data into the test environment

• When inserting test accounts into a production environment, plan thorough data preparation and cleanup logic

• Full-link stress testing also requires the same order of magnitude between test and production environments

Parameterized data volume

Parameterized data drives the variability of test inputs (user IDs, product IDs, search terms). Small parameter sets cause cache hits that inflate performance results.

Requirements:

• Maximize the parameterized data volume. If needed, clear caches or generate additional data programmatically

• Distribute parameterized data realistically. If a business operation has geographic distribution patterns (for example, region-specific inventory queries), reflect that in the test data

Progression strategy

Start with simpler test types and progress toward more complex ones. Run single-transaction benchmarks first to establish baselines, then move to mixed-transaction load testing, and then to stress and stability testing. At minimum, run the two required types: mixed-transaction load testing and mixed-transaction stability testing.

RPS mode and concurrency mode

PTS supports two load modes. Choose the mode that matches how your system receives real traffic:

• Open workload model (RPS mode -- recommended): You control the arrival rate of requests. Use this mode when your system faces uncontrolled incoming traffic, such as a public-facing web application. This is the most common scenario.

• Closed workload model (concurrency mode): You control the number of concurrent users. Use this mode when your system controls admission, such as a call center with a fixed number of agents.

Example: Two interfaces A and B have a production ratio of 1:4, with response times of 1 ms and 100 ms respectively.

RPS mode eliminates the need to calculate concurrency adjustments based on response time differences, giving more precise control over the traffic mix.

Why comprehensive monitoring matters

Monitoring during performance tests serves two purposes: real-time bottleneck detection and post-test root cause analysis. Without comprehensive metrics across all layers, identifying the source of a bottleneck becomes guesswork.

Metrics to monitor

Analysis focus by layer

Targeted analysis at each layer enables efficient tuning: fix the actual bottleneck rather than adding resources that have no impact.

Traffic simulation

PTS generates test traffic from Content Delivery Network (CDN) nodes across hundreds of cities and multiple carriers worldwide. This approach simulates realistic end-user access patterns, including geographic distribution and carrier-specific routing.

Scenario design

• Record scenarios using mainstream browser recording plug-ins, or orchestrate sequential and parallel API execution through a visual interface -- no coding required

• Parameterize requests using data files, built-in functions, string operations, and response extraction. PTS serves as a data factory that formats request parameters through simple encoding

• Maintain cookie and session state across API calls, with extensible instructions that support multi-form think time and traffic regulation

• Debug scenarios and individual APIs before running full-scale tests

Load control

• Two load modes: Concurrency mode and requests per second (RPS) mode

• Fast startup: Launch stress tests within minutes

• Real-time adjustment: Traffic changes take effect within seconds

• Pulse capability: Generate instantaneous spikes of millions of queries per second (QPS)

• Safety controls: Stop stress testing traffic immediately when needed

Reporting

PTS collects real-time data during each test, including concurrency, TPS, RT, and sampled logs for each API. Reports are generated automatically for post-test analysis.

Supported industries

PTS supports workloads across e-commerce, media, finance and insurance, logistics, advertising and marketing, and social networking.