MaxCompute (originally ODPS) is a Big Data processing platform used for batch structural data storage and processing, to provide massive data warehouse solutions and data modeling. MaxCompute2.0 is an upgrade to the original MaxCompute; the system's entire operating process has been optimized for performance. MaxCompute2.0 includes a next generation execution engine and compilation engine, and a cost-based optimization engine.
From testing, we can see that MaxCompute2.0 is better, faster, and stronger with greater functionality, more efficient use and better updating. MaxCompute2.0 doesn't only beat its predecessor, it is also as competitive as services provided by other Big Data vendors. According to performance metrics, MaxCompute2.0 offline computing performs better than its counterpart, Apache Hive2.0 on Tez, by more than 90%. Furthermore, the open ecosystem improves performance when dealing with SQL and SQL offline batch computing which respectively take up 80% of online computing and 50% of offline computing.
MaxCompute2.0 has developed a next generation execution engine. The new execution engine is based on Codegen and utilizes vectorized execution and cache-friendly algorithms. We can see that the new generation of execution engines boast remarkable advantages in performance over open source's next generation open source offline computing execution engine—Hive on Tez.
The chart below demonstrates operation times (in seconds) for both the next generation execution engine employed by MaxCompute2.0 and the community developed next generation offline execution engine Hive on Tez operating on the same amount of data using the same number of instances.
We can see that when operating with terabytes of data using hundreds of instances, Maxcompute2.0 consistently out-performs Hive2.0 on Tez (Hive2.0 is already using optimized settings). This includes:
1.Two times faster sum with group performance
2.Three times faster sort-merge join speeds
3.Two times faster hash join
4.50% faster streamline
MaxCompute2.0 has developed an all new parser and introduced a cost-based optimizer that is compatible with Hive syntax and semantics and utilizes optimizers based on a variety of rules (Rbo), introducing and developing accurate optimizer components based on statistical data and adding a collection of all new rules.
The following is end-to-end operation performance data from TPC-H benchmark comparing Maxcompute2.0 offline computing using its new compiler and cost-based optimizer with the community developed Hive2.0 on Tez under optimal operation settings:
Test environment:
1.Cluster scale: 30 test clusters, of which 20 are computing nodes
2.Machine configuration: 22core96GGigabit full dual-channel network121TB SATA hard drives on each node
3.Software versions: MaxCompute2.0Sp24rc5 / hive2.0onTez / MaxCompute1.0Sp23s14 / hive2.0onMr
4.Data size: 1TB (zlib compression)
In order to ensure data rationality, the performance test data is made up of stable values produced by several rounds of testing of each test case. The performance evaluation utilizes independent clusters, each cleared and recovered to its original settings prior to each test. Tests are not run in parallel, rather each round of tests is executed after the entire previous test set is complete.
From the comparison, we can conclude that:
1.MaxCompute2.0 offline computing is faster than its counterpart Hive2.0 on Tez by more than 90%.
2.MaxCompute2.0 is faster than Hive in executing more than 95% of benchmark SQL statements. We also analyzed the internal execution details. With the scheduling time and other time consumption deducted, MaxCompute2.0 actually outrivals Hive2.0 in execution performance by more than 114%.
3.MaxCompute2.0 performance is improved by 68% when compared to MaxCompute1.0.
4.MaxCompute2.0 outperforms Hive2.0 on Mr by 190% in terms of overall performance. Specifically, the performance of 77% of benchmark SQL statements is more than tripled.
Our goal is to have MaxCompute2.0 capable of all these features:
1.Compatible with community developed software, compatible with all Hive data types, follows SQL 2003, supports multidimensional grouping
2.Developed whole-stage code generation and, by condensing code that can slow down the entire search into one function, reduced the number of function calls and took advantage of CPU registers to store intermediate data and other drains on performance
3.More optimization rules as well as updates and upgrades to existing rules, default activation of join reordering, and support for range partitioning.
Apart from the huge performance improvements over the Apsara-based first generation execution engine, MaxCompute2.0 has also launched the following RBO and CBO execution performance optimizations:
1.Trimming rules: column trimming, partition trimming, sub-query trimming
2.Push-down/merge rules: predicate push-down
3.Deduplication rules: project deduplication, exchange deduplication, and sort deduplication
4.Constant folding/predicate derivation
5.Relation optimization: auto MapJoin, Skew Join; implement BroadcastHashJoin, ShuffleHashJoin, and MergeJoin; Join Reordering
6.Aggregate optimization: HashAggregate, SortedAggregate and deduplicate
7.Processing optimization: GroupBy push-down, exchange push-down, and sort push-down
Alibaba Cloud launched eight new Big Data and artificial intelligence (AI) products at the Mobile World Congress 2018 in Barcelona, Spain. These products, along with MaxCompute2.0, will meet the surging demand for powerful and reliable cloud computing services as well as advanced AI solutions among enterprises. Read more about the launch on the official press release.
2,605 posts | 747 followers
Follow- November 24, 2017
Alibaba Clouder - April 19, 2018
Alibaba Clouder - January 22, 2020
Alibaba Clouder - October 18, 2017
ApsaraDB - August 6, 2019
Alibaba Cloud Community - December 29, 2022
2,605 posts | 747 followers
Follow
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn More
MaxCompute
Conduct large-scale data warehousing with MaxCompute
Learn More
Realtime Compute
Realtime Compute offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn More
Batch Compute
Resource management and task scheduling for large-scale batch processing
Learn MoreMore Posts by Alibaba Clouder
