MaxFrame is a Pandas-compatible DataFrame API for MaxCompute. It lets you run Pandas-style data operations on large-scale datasets without being limited by local memory or compute resources. Use MaxFrame when your dataset exceeds single-machine memory or when your Pandas workflows are too slow to run locally.
To migrate from Pandas, change one import — MaxFrame handles the rest by distributing computation across the MaxCompute backend:
# Before (pandas)
import pandas as pd
# After (MaxFrame)
import maxframe.dataframe as mdMaxFrame uses lazy evaluation. Operations build a computation graph but do not run until you call.execute(). Chain multiple transformations freely, then call.execute()once to trigger computation.
Prerequisites
Before you begin, ensure that you have:
A MaxCompute project with the required permissions
Your Alibaba Cloud access key ID and secret access key (set as environment variables)
MaxFrame installed in your Python environment
Sample dataset
This guide uses the maxframe_ml_100k_users table from the MaxCompute public dataset. The table is available at BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users — no import needed.
Initialize a session
Before running any MaxFrame job, initialize a session at the entry point of your code. All subsequent data processing goes through this session, which communicates with the MaxCompute backend.
import os
from maxframe import new_session
from odps import ODPS
# Initialize MaxCompute with your credentials.
o = ODPS(
os.getenv('ALIBABA_CLOUD_ACCESS_KEY_ID'),
os.getenv('ALIBABA_CLOUD_ACCESS_KEY_SECRET'),
project='your-default-project',
endpoint='your-end-point',
)
# Start the MaxFrame session.
new_session(odps_entry=o)For parameter details, see MaxFrame-specific APIs.
Create a DataFrame
Use read_odps_table or read_odps_query to load data from MaxCompute into a MaxFrame DataFrame. You can also create a DataFrame from local data for quick testing.
Read from a MaxCompute table
read_odps_table reads a MaxCompute table and returns a DataFrame.
import maxframe.dataframe as md
# Read all columns from the table.
df = md.read_odps_table('BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users')Use index_col to set a column as the DataFrame index, or columns to select a subset of columns:
# Set user_id as the index.
df = md.read_odps_table(
'BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users',
index_col='user_id'
)
# Select specific columns.
df = md.read_odps_table(
'BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users',
columns=['user_id', 'age', 'sex']
)For full parameter details, see MaxFrame-specific APIs.
Read from a SQL query
Run a SQL query and use the result as a DataFrame:
import maxframe.dataframe as md
df = md.read_odps_query(
'select user_id, age, sex FROM `BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users`'
)
# Optionally set the index column.
df = md.read_odps_query(
'select user_id, age, sex FROM `BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users`',
index_col='user_id'
)For full parameter details, see MaxFrame-specific APIs.
Create from local data
For quick local testing, create a DataFrame directly from a Python dict:
import maxframe.dataframe as md
d = {'col1': [1, 2], 'col2': [3, 4]}
df = md.DataFrame(data=d)
print(df.execute().fetch())
# Result:
# col1 col2
# 0 1 3
# 1 2 4Process data
MaxFrame supports the same data operations as Pandas. All operations use lazy evaluation — chain as many as you need, then call .execute() once to trigger computation.
Mathematical operations
Apply arithmetic operations column-wise. All standard operators (+, -, *, /) work as in Pandas.
import maxframe.dataframe as md
df = md.DataFrame({'angles': [0, 3, 4],
'degrees': [360, 180, 360]},
index=['circle', 'triangle', 'rectangle'])
# Addition
print((df + 1).execute().fetch())
# Result:
# angles degrees
# circle 1 361
# triangle 4 181
# rectangle 5 361
# Multiplication
print((df * 2).execute().fetch())
# Result:
# angles degrees
# circle 0 720
# triangle 6 360
# rectangle 8 720
# Division
print((df / 2).execute().fetch())
# Result:
# angles degrees
# circle 0.0 180
# triangle 1.5 90
# rectangle 2.0 180
# Exponentiation
print((df ** 2).execute().fetch())
# Result:
# angles degrees
# circle 0 129600
# triangle 9 32400
# rectangle 16 129600For more information, see Binary operator functions.
Filtering, projection, and sampling
Filtering
Select rows that match a condition:
import maxframe.dataframe as md
df = md.read_odps_table('BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users')
# Keep only rows where age > 18.
filtered_df = df[df['age'] > 18]
print(filtered_df.execute().fetch())
# Result:
# user_id age sex occupation zip_code
# 0 1 24 M technician 85711
# 1 2 53 F other 94043
# ...Projection
Select a subset of columns:
import maxframe.dataframe as md
df = md.read_odps_table('BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users')
# Select specific columns.
projected_df = df[['user_id', 'age']]
print(projected_df.execute().fetch())
# Result:
# user_id age
# 0 1 24
# 1 2 53
# ...Sampling
Draw a random sample from a DataFrame:
import maxframe.dataframe as md
df = md.DataFrame({'num_legs': [2, 4, 8, 0],
'num_wings': [2, 0, 0, 0],
'num_specimen_seen': [10, 2, 1, 8]},
index=['falcon', 'dog', 'spider', 'fish'])
print(df['num_legs'].sample(n=3, random_state=1).execute())
# Result:
# falcon 2
# fish 0
# dog 4
# Name: num_legs, dtype: int64For more information, see Reindexing / selection / label manipulation.
Sorting
Sort a DataFrame by one or more columns using sort_values:
import maxframe.dataframe as md
import numpy as np
df = md.DataFrame({
'col1': ['A', 'A', 'B', np.nan, 'D', 'C'],
'col2': [2, 1, 9, 8, 7, 4],
'col3': [0, 1, 9, 4, 2, 3],
})
# Sort by col1.
res = df.sort_values(by=['col1']).execute()
print(res.fetch())
# Result:
# col1 col2 col3
# 0 A 2 0
# 1 A 1 1
# 2 B 9 9
# 5 C 4 3
# 4 D 7 2
# 3 None 8 4
# Sort by col1, then col2.
res = df.sort_values(by=['col1', 'col2']).execute()
print(res.fetch())
# Result:
# col1 col2 col3
# 1 A 1 1
# 0 A 2 0
# 2 B 9 9
# 5 C 4 3
# 4 D 7 2
# 3 None 8 4For more information, see Reshaping / sorting / transposing.
Join, merge, and concatenate
Combine DataFrames horizontally (join/merge) or vertically (concatenate).
import maxframe.dataframe as md
df1 = md.DataFrame({'key': ['K0', 'K1', 'K2'],
'A': ['A0', 'A1', 'A2'],
'B': ['B0', 'B1', 'B2']})
df2 = md.DataFrame({'key': ['K0', 'K1', 'K2'],
'C': ['C0', 'C1', 'C2'],
'D': ['D0', 'D1', 'D2']})
# Merge on a key column.
result = md.merge(df1, df2, on='key')
print(result.execute().fetch())
# Result:
# key A B C D
# 0 K0 A0 B0 C0 D0
# 1 K1 A1 B1 C1 D1
# 2 K2 A2 B2 C2 D2
# Concatenate vertically.
result = md.concat([df1, df2])
print(result.execute().fetch())For more information, see Combining / joining / merging.
Aggregation and user-defined functions (UDFs)
Aggregation
Summarize data across groups or columns with aggregation functions:
import maxframe.dataframe as md
import numpy as np
df = md.DataFrame([[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[np.nan, np.nan, np.nan]],
columns=['A', 'B', 'C'])
# Apply multiple aggregation functions.
print(df.agg(['sum', 'min']).execute().fetch())
# Result:
# A B C
# min 1.0 2.0 3.0
# sum 12.0 15.0 18.0
# Apply different functions to different columns.
df.agg({'A': ['sum', 'min'], 'B': ['min', 'max']}).execute().fetch()
# Result:
# A B
# max NaN 8.0
# min 1.0 2.0
# sum 12.0 NaNUDFs
MaxFrame supports UDFs through transform and apply. Use UDFs for custom logic that built-in operators don't cover.
Before running a UDF, declare the common image by setting config.options.sql.settings before calling new_session.
All UDF examples use this config block:
from maxframe import config
config.options.sql.settings = {
"odps.session.image": "common",
"odps.sql.type.system.odps2": "true"
}
session = new_session(o)Use `transform` to apply a UDF column-wise (output shape matches input):
import maxframe.dataframe as md
df = md.DataFrame({'A': range(3), 'B': range(1, 4)})
print(df.transform(lambda x: x + 1).execute().fetch())
# Result:
# A B
# 0 1 2
# 1 2 3
# 2 3 4Use `apply` when the output column count differs from the input:
import maxframe.dataframe as md
import numpy as np
def simple(row):
row['is_man'] = row['sex'] == "man"
return row
df = md.read_odps_table('BIGDATA_PUBLIC_DATASET.data_science.maxframe_ml_100k_users')
new_dtypes = df.dtypes.copy()
new_dtypes["is_man"] = np.dtype(np.bool_)
df.apply(
simple,
axis=1,
result_type="expand",
output_type="dataframe",
dtypes=new_dtypes
).execute().fetch()
# Result:
# user_id age sex occupation zip_code is_man
# 0 1 24 M technician 85711 False
# 1 2 53 F other 94043 False
# ...
# [943 rows x 6 columns]For more information, see Function application / GroupBy / window.
Store results
After processing, save results to a MaxCompute table with to_odps_table.
# Write results to a MaxCompute table.
filtered_df.to_odps_table('<table_name>')
# Set a lifecycle (in days) for the result table.
filtered_df.to_odps_table('<table_name>', lifecycle=7)Parameter | Description |
| Destination table name. If the table does not exist, MaxCompute creates it automatically. If it already exists, data is appended by default. |
| Number of days to retain the result table. |
| Set to |
For more information, see MaxFrame-specific APIs.
Execute and retrieve results
MaxFrame uses lazy evaluation — operations build a computation graph but do not run until you call .execute(). Chain .execute().fetch() to trigger execution and retrieve results.
# Execute the computation and fetch results.
data = filtered_df.execute().fetch()
print(data)Method | Description |
| Submits the computation graph to MaxCompute and waits for completion. |
| Retrieves a portion of the result data after execution. |
| Triggers execution and returns results in one step. |
For more information, see MaxFrame-specific APIs and MaxFrame-specific APIs.