MaxCompute2.0 Performance Evaluation
Created#More Posted time:Nov 3, 2016 13:24 PM
MaxCompute2.0 (former ODPS): Through performance evaluation, offline computing by MaxCompute2.0 outperforms that by Hive2.0 on Tez, a similar product, by more than 90%; MaxCompute2.0 has made full-process improvement on the performance from the new generation execution engine to compilation engine as well as to the cost-based optimizer.
This evaluation focuses on the performance comparison between the released MaxCompute2.0 and other offline computing products and between MaxCompute2.0 and the online stable version. We can see from the test that MaxCompute2.0 has more powerful functions, more efficient use, updating and releasing functions and an open ecosystem. Meanwhile, it boasts great performance improvements in SQL computing that accounts for more than 80% of online jobs, and in offline SQL batch computing that accounts for about 50% of SQL computing.
I. A new generation of execution engine
MaxCompute2.0 develops a new generation of execution engine. The new execution engine is based on Codegen and adopts vectorized execution and cache-friendly algorithms. We can see that the next generation of execution engine boasts remarkable advantages in performance over Hive on Tez, the next generation offline computing execution engine in the open-source community.
We can record the time used by MaxCompute2.0 and Hive on Tez for processing the same data size with the same number of instances (the unit is seconds). The table below can reflect the performance of the new generation execution engines.
We can see that when processing terabytes of data on hundreds of instances, the average execution performance of MaxCompute2.0 for the same data size on the same number of instances is higher than Hive2.0OnTez (Hive2.0 has been tuned to the optimal mode):
1- sum with group performance of MaxCompute2.0 is doubling that of Hive2.0OnTez
2-sort-merge join performance is more than tripled
3-hashjoin performance is more than doubled
4-stremline performance is improved by more than 50%
II. New compilation engine and cost-based optimizer
MaxCompute2.0 has developed a new parser and introduced the development of cost-based optimizer. While staying compatible with Hive syntax and semantics and developing and applying various rule-based optimizers (RBO), MaxCompute2.0 has introduced and developed a more accurate optimizer component under the guidance of statistic data, and added new optimizing rules.
According to the evaluations on end-to-end execution performance of the new compilation engine and cost-based optimizer, the offline computing TPC-H benchmark data of MaxCompute2.0 and Hive2.0 on Tez in the optimal mode is as follows:
1. Cluster scale: 30 servers, with 20 computing nodes
2. Server configuration: 22-core\96G\Gigabit full duplex network\12 1TB SATA disks at each node
3. Software version: MaxCompute2.0Sp24rc5/hive2.0onTez/MaxCompute1.0Sp23s14/hive2.0onMr
4. Data size: 1TB (zlib compressed)
In order to ensure data rationality, the performance test data is the stable values after several rounds of tests of each test case. Independent clusters are established for the performance evaluation and each cluster is cleared and recovered to the initial environment before tests. The several rounds of tests are not continuous. The next round of tests is executed after the entire test sets are completed.
The comparison data in the figure is:
1. MaxCompute2.0 offline computing is faster than its counterpart Hive2.0 on Tez by more than 90%.
2. MaxCompute2.0 is faster than Hive in executing more than 95% of benchmark SQL statements. We also analyzed the internal execution details. With the scheduling time and other time consumption deducted, MaxCompute2.0 actually outrivals Hive2.0 in execution performance by more than 114%.
3. MaxCompute2.0 improves the performance by 68% more than MaxCompute1.0.
4. MaxCompute2.0 outperforms Hive2.0 on Mr by 190% in terms of overall performance. In specific, the performance of 77% of benchmark SQL statements is more than tripled.
III. MaxCompute2.0 performance enhancements and expectations for future performance evolution
1. Be compatible with community products and all Hive data types; comply with SQL 2003 and support multi-dimensional grouping
2. The whole-stage code generation will be researched and developed, that is, eliminate performance consumption from calling functions and from storing intermediate data into the CPU register by optimizing the code that impedes the overall query speed to an independent function during runtime
3. More optimization rules, updates and upgrades to the existing optimization rules. Join Reordering will be enabled by default and range partitioning will be supported
Apart from the huge performance improvement over the Apsara-based Generation 1 execution engine, MaxCompute2.0 has launched the following RBO and CBO optimizations in execution performance:
1- Trimming rules: column trimming, partition trimming, sub-query trimming
2- Push-down/merge rules: predicate push-down
3- Deduplication rules: project deduplication, exchange deduplication, and sort deduplication
4- Constant folding/predicate derivation
5- Relation optimization: auto MapJoin, Skew Join; implement BroadcastHashJoin, ShuffleHashJoin, and MergeJoin; Join Reordering
6- Aggregate optimization: HashAggregate, SortedAggregate and deduplicate
7- Processing optimization: GroupBy push-down, exchange push-down, and sort push-down
[Charlene edited the post at Nov 4, 2016 13:15 PM]