TPC-DS, performance testing - MaxCompute - Alibaba Cloud Documentation Center

MaxCompute supports data query scenarios at scales ranging from terabytes (TB) to petabytes (PB) and exabytes (EB), offering significant performance advantages in the industry. This topic uses MaxCompute’s public dataset and test tools to validate MaxCompute performance through the TPC-DS big data benchmark. MaxCompute provides two performance testing methods: Method 1 (recommended) uses the new MaxQA Query Acceleration engine for TPC-DS testing, and Method 2 uses the MCQA engine for TPC-DS testing (MCQA is no longer open to new users; existing users may refer to this method).

Preparations

Prepare your environment.
- Before running a TPC-DS test, activate MaxCompute and create a project. For details, see Appendix: Project configuration parameters.
- Method 1 (recommended): Activate MaxQA (MaxCompute Query Acceleration 2.0). This feature is currently in public preview (click the public preview link to join; check the public preview schedule for regional availability).
  - See Overview of MaxQA Query Acceleration to activate a subscription instance and create an interactive quota group.
- Method 2 (no longer open to new users): Activate MCQA (MaxCompute Query Acceleration 1.0), available only to existing MCQA users. See MCQA Query Acceleration (no longer open to new users) to activate a subscription instance.
Prepare the test tool.
To simplify testing, MaxCompute provides an automated TPC-DS performance testing tool that helps you run TPC-DS tests quickly and generate results automatically.
Important
The test tool currently supports only Linux environments and requires JDK version 1.7 or later.
Click mc_tpcds_benchmark to download the tool package, then run the following command on your Linux server to extract it.
```
unzip mc_tpcds_benchmark.zip
```
The extracted directory structure is as follows:
```
.
|_t1c7039e3-2a1d-451b-bfda-d14c49016243-tpc-ds-tool.zip
|_config
|_init_tools.sh
|_load_table.sh
|_logs
|_odps_clt
|_patches
|_pt.sh
|_queries_1
|_queries_1.quality
|_queries_10
|_queries_100
|_queries_1000
|_queries_10000
|_queries_100000
|_querygen.sh
|_results
|_run_stream.sh
|_run_stream.sh.offline
|_sqls
|_start_session_only.sh
|_start_session.sql
|_start_session.sql_tmp
|_tools_file
|_tt.sh
|_v2.10.1rc3
```

Understand the test dataset.

MaxCompute provides a public dataset so you do not need to prepare test data yourself. All data is stored in the MaxCompute public project BIGDATA_PUBLIC_DATASET. For details, see Overview of public datasets.

The TPC-DS test dataset comes in four sizes: 10 GB, 100 GB, 1 TB, and 10 TB. Choose the size that fits your testing needs. Details are shown below:

Category	Description	Dataset name	Schema name
TPC-DS	TPC-DS is a decision support benchmark that models common aspects of decision support systems, including queries and data maintenance, enabling new technologies such as big data systems to undergo standardized benchmarking.	TPC-DS 10GB performance test set TPC-DS 100GB performance test set TPC-DS 1TB performance test set TPC-DS 10TB performance test set	tpcds_10g tpcds_100g tpcds_1t tpcds_10t

Testing procedure

Modify the test tool configuration file

Go to the mc_tpcds_benchmark directory of the extracted test tool and edit the config file. Because the tool supports both MaxQA and MCQA modes, some configuration settings differ slightly between the two beyond the basic settings.

Basic configuration

Configuration item	Description	Value
ODPS_CLT_CMD	Absolute path to the MaxCompute client executable file. The client included in this toolkit is located under odps_clt in the working directory. Update this setting accordingly. For details, see Connect using the local client (odpscmd).	Example: /xxxxx/mc_tpcds_benchmark/odps_clt/bin/odpscmd
PROJECT	The MaxCompute project used for testing.	Example: tpcds_test
SF	Data scale for the TPC-DS test. The unit is GB. 1 means 1 GB, and 1000 means 1 TB. Adjust this value as needed.	Default: 1000

Differences between MCQA and MaxQA modes

Configuration item	Description	MCQA value	MaxQA value
ODPS_CLT_CMD	Absolute path to the MaxCompute client executable file. The client included in this toolkit is located under odps_clt in the working directory. Update this setting accordingly. For details, see Connect using the local client (odpscmd).	Example: /xxxxx/mc_tpcds_benchmark/odps_clt/bin/odpscmd	Example: /xxxxx/mc_tpcds_benchmark/odps_clt/bin/odpscmd
PROJECT	The MaxCompute project used for testing.	Example: tpcds_test	Example: tpcds_test
SF	Data scale for the TPC-DS test. The unit is GB. 1 means 1 GB, and 1000 means 1 TB. Adjust this value as needed.	Default: 1000. MaxCompute public datasets currently include four sizes: 10G, 100G, 1T, and 10T. Therefore, set SF to 10, 100, 1000, or 10000.	Default: 1000. MaxCompute public datasets currently include four sizes: 10G, 100G, 1T, and 10T. Therefore, set SF to 10, 100, 1000, or 10000.
MODE	Set the mode for this benchmark run.	MCQA	MaxQA
MAXQA_QUOTA_NAME	Name of the MaxQA interactive quota group used for testing. You can find this in the Quota Management page of the MaxCompute console. Be sure to enter the quota alias (the name you assigned).	N/A	Example: maxqa_test_quota
SQL_FLAGS	Built-in MaxCompute SQL flags. Do not modify.	`set odps.sql.session.result.cache.enable=false`: Disable result caching in MCQA mode to ensure each query runs independently. `set odps.sql.allow.cartesian=true`: Enable Cartesian product computation in SQL. `set odps.sql.session.query.timeout=600`: Set Fuxi job timeout in MCQA mode.	`set odps.sql.mcqa2.result.cache.enable=false`: Disable result caching in MaxQA mode to ensure each query runs independently. `set odps.sql.allow.cartesian=true`: Enable Cartesian product computation in SQL.

Run the test

In the mc_tpcds_benchmark directory, run the following command to start the TPC-DS test:

nohup sh pt.sh > pt.log 2>&1 &

After successful execution, the tool automatically generates a pt.log log file in the mc_tpcds_benchmark directory. Run the following command to view detailed task logs:

tail -f pt.log

Monitor MaxCompute task execution

Log in to the MaxCompute console and select a region in the upper-left corner.
In the left-side navigation pane, choose Observation O&M > Jobs.
On the Job O&M page, check task execution status. Click LogView in the Actions column of the target job to view detailed information. For more information, see Job O&M.

View test results

After successful task execution, the tool automatically generates a test result file named console_test_result.csv in the mc_tpcds_benchmark directory. This file contains the total test duration, execution time for each query, and corresponding Logview links.