Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog MaxCompute Clustering Optimization Recommendation: Save 2 PB of Shuffle and 7,000+ CU-hours Daily!

MaxCompute Clustering Optimization Recommendation: Save 2 PB of Shuffle and 7,000+ CU-hours Daily!

This article introduces a new MaxCompute feature that intelligently recommends data clustering optimizations to dramatically reduce shuffle traffic and daily compute costs.

By Alibaba Cloud MaxCompute Team

A Powerful Tool for Shuffle Optimization | Clustering Optimization Recommendation

In MaxCompute’s daily exabyte-scale computing environment, shuffle traffic generated by operators like Join, Group By, and Window accounts for over 60% of total network transmission—making it a primary driver of big data compute costs. For example, one internal Alibaba business generates 2 PB of shuffle data and consumes over 7,000 CU-hours per day—a figure that represents just the tip of the iceberg.

The MaxCompute Hash Clustering table feature reorganizes and sorts data by defining shuffle and sort attributes. This significantly reduces I/O consumption in downstream processing pipelines, accelerates query and computation tasks, and ultimately improves job efficiency while lowering resource costs.

However, many tables are not initially configured with Hash Cluster. As business scales and data workflows grow more complex, retroactively applying data governance becomes challenging, requiring deep historical analysis to make informed decisions.

To help users optimize their data pipelines more efficiently, MaxCompute has launched the Clustering Optimization Recommendation feature. Based on 31 days of historical run data, this feature automatically identifies the globally optimal Hash Cluster Key each day. For large-scale shuffle scenarios exceeding 10 GB, this capability delivers substantial cost savings.

Proven Results | Technical Deep Dive

The Clustering Optimization Recommendation feature is already widely adopted across Alibaba, delivering measurable performance improvements. We believe that as more businesses adopt this solution, they will unlock significant speedups and maximize their data processing potential.

1

So, what makes this feature so effective? Let’s explore its core advantages:

  • Global Directed Acyclic Graph (DAG) awareness: Analyzes shuffle dependency graphs across thousands of jobs simultaneously, providing a holistic view of data flow patterns to precisely identify optimization opportunities.
  • Dynamic skew detection: Proactively identifies hot spot keys to prevent situations where optimization inadvertently degrades performance, ensuring stability and reliability.
  • Intelligent benefit assessment: Uses smart algorithms to recommend changes only for tables with "high shuffle + low risk," avoiding unnecessary modifications and ensuring recommendations are both efficient and trustworthy.
  • One-click script generation: Automatically generates ALTER TABLE statements and provides rollback plans, dramatically simplifying implementation and enabling fast, safe optimization.

This feature doesn’t just reduce costs—it also accelerates query speeds and boosts resource utilization across your entire data ecosystem.

2

Quick Start: Use the Clustering Optimization Recommendation

This feature is now live on the Alibaba Cloud MaxCompute console. With just three simple steps, you can discover, apply, and validate optimizations. View the recommendation list and apply the recommendations.

Step 1: View recommendations and apply

1.  Log in to the MaxCompute console → Intelligent Optimization → Data Layout Optimization → Clustering Optimization.

3

Review the list of recommended optimizations based on estimated benefits. You’ll see:

  • Which project/table can be optimized
  • Which column(s) to use as ClusterKey, SortKey, and bucket count
  • Estimated shuffle volume savings

2.  Select a table and click "Go to Optimize" for a detailed plan.

4

View a list of related jobs (read, write, full read/write) expected to benefit.

3.  Click "Apply Recommendations" to generate the ALTER TABLE statement and rollback script. Click "Confirm Application" to convert the table into a Hash Cluster table.

5

Step 2: Monitor optimization benefits

On the Clustering Optimization tab:

  • Select "Actual Benefits" and choose an analysis period.

6

  • View:

    • Total benefits: Number of benefited jobs, CU-hour savings, and shuffle volume reduction.
    • Optimized list: A detailed list of modified tables, including modification time, number of benefited jobs, and savings in billable hours, CU-hours, and shuffle volume.
    • Optimization details: Drill down into performance gains for specific tables.

7

For more details, see Clustering optimization recommendations >>

More MaxCompute Optimization Recommendations

MaxCompute continues to innovate with a suite of intelligent optimization tools. Future enhancements include:

• Optimizer: Automatically merges cluster keys in CASE WHEN / COALESCE scenarios.

• Intelligent Data Warehouse: AutoMV, compute configuration optimization recommendations, tiered storage optimization recommendations. Future releases will combine Z-Order and Data Skipping for composite index recommendations.

• Real-time recommendations: Pushes next-hop optimization suggestions immediately after a job completes.

MaxCompute Big Data Alibaba Cloud MaxCompute Clustering Optimization Clustering Optimization Recommendation Intelligent optimization tools Optimization Recommendations
0 1 0
Share on

Read previous post:

S&P Global Partners with Alibaba Cloud to Accelerate AI Innovation in China with First Local Access to AI-Ready Commodity Data

Read next post:

Alibaba Open-Sources Tongyi DeepResearch LLM, Partners with S&P Global to Deliver AI Intelligence to Chinese Customers

Alibaba Cloud Community

1,276 posts | 453 followers

You may also like

Comments

Alibaba Cloud Community

1,276 posts | 453 followers

Related Products

More Posts by Alibaba Cloud Community

See All