MaxCompute Clustering Optimization Recommendation: Save 2 PB of Shuffle and 7,000+ CU-hours Daily!
By Alibaba Cloud MaxCompute Team
A Powerful Tool for Shuffle Optimization | Clustering Optimization Recommendation
In MaxCompute’s daily exabyte-scale computing environment, shuffle traffic generated by operators like Join, Group By, and Window accounts for over 60% of total network transmission—making it a primary driver of big data compute costs. For example, one internal Alibaba business generates 2 PB of shuffle data and consumes over 7,000 CU-hours per day—a figure that represents just the tip of the iceberg.
The MaxCompute Hash Clustering table feature reorganizes and sorts data by defining shuffle and sort attributes. This significantly reduces I/O consumption in downstream processing pipelines, accelerates query and computation tasks, and ultimately improves job efficiency while lowering resource costs.
However, many tables are not initially configured with Hash Cluster. As business scales and data workflows grow more complex, retroactively applying data governance becomes challenging, requiring deep historical analysis to make informed decisions.
To help users optimize their data pipelines more efficiently, MaxCompute has launched the Clustering Optimization Recommendation feature. Based on 31 days of historical run data, this feature automatically identifies the globally optimal Hash Cluster Key each day. For large-scale shuffle scenarios exceeding 10 GB, this capability delivers substantial cost savings.
Proven Results | Technical Deep Dive
The Clustering Optimization Recommendation feature is already widely adopted across Alibaba, delivering measurable performance improvements. We believe that as more businesses adopt this solution, they will unlock significant speedups and maximize their data processing potential.

So, what makes this feature so effective? Let’s explore its core advantages:
- Global Directed Acyclic Graph (DAG) awareness: Analyzes shuffle dependency graphs across thousands of jobs simultaneously, providing a holistic view of data flow patterns to precisely identify optimization opportunities.
- Dynamic skew detection: Proactively identifies hot spot keys to prevent situations where optimization inadvertently degrades performance, ensuring stability and reliability.
- Intelligent benefit assessment: Uses smart algorithms to recommend changes only for tables with "high shuffle + low risk," avoiding unnecessary modifications and ensuring recommendations are both efficient and trustworthy.
- One-click script generation: Automatically generates ALTER TABLE statements and provides rollback plans, dramatically simplifying implementation and enabling fast, safe optimization.
This feature doesn’t just reduce costs—it also accelerates query speeds and boosts resource utilization across your entire data ecosystem.

Quick Start: Use the Clustering Optimization Recommendation
This feature is now live on the Alibaba Cloud MaxCompute console. With just three simple steps, you can discover, apply, and validate optimizations. View the recommendation list and apply the recommendations.
Step 1: View recommendations and apply
1. Log in to the MaxCompute console → Intelligent Optimization → Data Layout Optimization → Clustering Optimization.

Review the list of recommended optimizations based on estimated benefits. You’ll see:
- Which project/table can be optimized
- Which column(s) to use as ClusterKey, SortKey, and bucket count
- Estimated shuffle volume savings
2. Select a table and click "Go to Optimize" for a detailed plan.

View a list of related jobs (read, write, full read/write) expected to benefit.
3. Click "Apply Recommendations" to generate the ALTER TABLE statement and rollback script. Click "Confirm Application" to convert the table into a Hash Cluster table.

Step 2: Monitor optimization benefits
On the Clustering Optimization tab:
- Select "Actual Benefits" and choose an analysis period.

-
View:
- Total benefits: Number of benefited jobs, CU-hour savings, and shuffle volume reduction.
- Optimized list: A detailed list of modified tables, including modification time, number of benefited jobs, and savings in billable hours, CU-hours, and shuffle volume.
- Optimization details: Drill down into performance gains for specific tables.

For more details, see Clustering optimization recommendations >>
More MaxCompute Optimization Recommendations
MaxCompute continues to innovate with a suite of intelligent optimization tools. Future enhancements include:
• Optimizer: Automatically merges cluster keys in CASE WHEN / COALESCE scenarios.
• Intelligent Data Warehouse: AutoMV, compute configuration optimization recommendations, tiered storage optimization recommendations. Future releases will combine Z-Order and Data Skipping for composite index recommendations.
• Real-time recommendations: Pushes next-hop optimization suggestions immediately after a job completes.
Read previous post:
Read next post:
-
-
Alibaba Cloud Community
1,605 posts | 526 followers
Follow
Comments
-
-
Alibaba Cloud Community
1,605 posts | 526 followers
Follow
Related Products
-
MaxCompute
Conduct large-scale data warehousing with MaxCompute
Learn More -
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More -
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn More -
Financial Services Solutions
Alibaba Cloud equips financial services providers with professional solutions with high scalability and high availability features.
Learn More
More Posts by Alibaba Cloud Community
- The Easiest Way to Try the Latest Models in One Place: Token Plan Starting at $3 for the First Month
- 一個平台輕鬆試用最新 AI 模型:Token Plan 首月低至 $3
- Alibaba Cloud Advances AI Adoption in Malaysia with Nationwide Youth Contest and Industry Collaborations
- Alibaba Cloud Unveils New AI Initiatives at Hong Kong Summit to Advance the City’s AI Ecosystem
- QoderWake: Digital Workers Take the Initiative — No Need to @ Every Time
- What Is Model Thinking Effort? Should You Choose Low, High, or Maximum?
- High-Risk PHP and Node.js Vulnerabilities: A Scan Costs Pennies. A Miss Could Cost a Fortune.
- Alibaba Unveils Qwen4-Max: New AI Teammate for Work, Creativity, and Discovery
- Alibaba Brings Cloud and AI to the First Youth Olympic Games in Africa
- Introducing Qoder Lite: Start creating with Qoder for $10 a month