All Products
Search

This Product

    MaxCompute:MaxFrame: The distributed AI compute engine

    Document Center

    MaxCompute:MaxFrame: The distributed AI compute engine

    Last Updated:Jun 02, 2026

    Overview

    MaxFrame is a distributed computing framework from Alibaba Cloud MaxCompute with a Python programming interface. It solves two key challenges in traditional Python data processing: performance bottlenecks and inefficient data movement. MaxFrame lets you process and analyze petabyte-scale data directly on MaxCompute for visual data exploration, scientific computing, machine learning, and AI development.

    Use cases

    Interactive data exploration

    Explore, manipulate, and visualize massive datasets in real time without memory limits, just as you would in a local Jupyter Notebook.

    Large-scale data preprocessing (ETL)

    Replace complex SQL+UDF logic with expressive, maintainable Python for multi-terabyte data cleansing, format conversion, and feature engineering—with the performance of distributed execution.

    AI and machine learning

    MaxFrame unifies data processing and model training. Prepare training data efficiently and use the image feature to import libraries such as Scikit-learn and XGBoost for end-to-end AI workflows.

    Usage notes

    Supported regions

    China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), China (Hong Kong), Japan (Tokyo), Singapore, Indonesia (Jakarta), Germany (Frankfurt), US (Silicon Valley), and US (Virginia).

    Supported environments

    • Local Python development environment.

    • MaxCompute Notebook.

    • DataWorks Notebook.

    • DataWorks Data Development PyODPS 3 task nodes.

    Billing

    MaxFrame bills based on compute resource usage per job. It supports and subscription billing.

    • Subscription: Jobs consume purchased resource group quota at no additional charge.

    Analyze MaxCompute bill and usage details.

    Core advantages

    MaxFrame offers key advantages over other Python tools for MaxCompute:

    • Pandas-compatible API: Migrate existing Pandas code to MaxCompute with minimal changes, reducing learning and migration costs.

    • Server-side distributed execution: Jobs run within the MaxCompute cluster without pulling data to your local machine, eliminating client memory bottlenecks and enabling petabyte-scale processing.

    • Elastic computing resources: Built on MaxCompute serverless architecture, MaxFrame allocates resources on demand. Process data at any scale without managing clusters.

    • Simplified development environment: Built-in Python 3.7 and Python 3.11 environments with pre-installed libraries such as Pandas and XGBoost. Add third-party dependencies through simple annotations—no manual packaging or UDF uploads required.

    The following table compares MaxFrame with other Python development tools:

    Comparison Item

    MaxFrame

    PyODPS

    Mars

    SQL+UDF

    Development API

    Compatible with Pandas.

    Syntax and API differ significantly from Pandas DataFrame.

    Requires using two sets of APIs: SQL and Python.

    Data processing

    Data stays on the server at runtime—no local transfer needed, improving execution efficiency.

    The to_pandas method pulls data locally for computation.

    Distributed execution for some operators only.

    Requires cluster creation during initialization; slow with low stability.

    Distributed jobs via MaxCompute SQL.

    Computing resources

    Not limited by local resources, breaking single-machine Python bottlenecks.

    Limited by local resources.

    Limited by resource size. Workers, CPU, and memory must be specified.

    Elastic computing for SQL jobs via MaxCompute serverless.

    Development experience

    Out-of-the-box interactive development and offline scheduling. Common libraries built-in. Manage dependencies via annotations—no manual packaging.

    Out-of-the-box interactive development and offline scheduling.

    Requires setting up a runtime environment and launching a Mars cluster.

    Python UDF dependencies must be manually packaged and uploaded.

    How it works

    MaxFrame abstracts distributed computing complexity. The workflow is:

    1. Code submission: Write and run Python code in a client such as a Notebook. The MaxFrame SDK captures and submits the code to MaxCompute.

    2. Parsing and optimization: The MaxCompute execution engine parses the syntax, optimizes the logic, and transforms the job into a parallel physical plan.

    3. Distributed execution: The optimized task is distributed across compute nodes in the MaxCompute cluster, which read data and compute in parallel.

    4. Result return: Results are aggregated and returned to your client.

    References