MaxCompute:MaxFrame multimodal data processing operators

Challenges

Processing multimodal data (images, audio) at scale for AI model training presents several challenges:

• Low throughput: Single-machine processing cannot handle large-scale multimodal data efficiently.

• Complex ops: Building and maintaining dedicated processing clusters adds operational overhead.

• Fragmented pipelines: Structured data processing, multimodal processing, and AI inference run on separate systems, increasing development complexity.

Key features

MaxFrame provides built-in multimodal operators that unify large-scale data processing with AI Function offline inference in a single pipeline.

• Distributed processing

  Processes massive multimodal datasets in parallel on the MaxCompute distributed engine.

• Native OSS integration

  Reads multimodal data directly from OSS without data migration.

• Rich operator set

  Covers image operations (decode, properties, resize, crop, format conversion) and audio operations (decode, transcribe, language detection, VAD).

• Declarative API

  Pandas-like API with chainable calls for a minimal learning curve.

• Secure authentication

  Accesses OSS through RAM roles (role_arn) — no hard-coded AK/SK required.

Use cases

• Image data cleaning: Filter invalid images and select those that meet size or format requirements.

• Image preprocessing: Prepare training data through resize, crop, and format conversion.

• Audio transcription: Batch-transcribe audio to text with multi-language support.

• Multimodal dataset construction: Build large-scale image and audio datasets for model training.

Example 1: Extract and filter image properties

Download images from OSS in batch, extract properties, filter by criteria, and write results to a table.

from maxframe import dataframe as md

# Read data table containing image OSS paths
df = md.read_odps_table("image_src_table")

# Download → decode → property extraction (chained calls)
df["img_bytes"] = df["oss_path"].url.download(
    storage_options={"role_arn": ROLE_ARN}
)
df["img_obj"] = df["img_bytes"].image.decode()

df["width"]  = df["img_obj"].image.width
df["height"] = df["img_obj"].image.height
df["size"]   = df["img_obj"].image.size
df["format"] = df["img_obj"].image.format
df["mode"]   = df["img_obj"].image.mode

# Filter by width and height range
df_filtered = df[
    df["width"].between(1000, 5000) &
    df["height"].between(2000, 6000)
]

# Write to result table
df_filtered[
    ["id", "oss_path", "width", "height", "size", "format", "mode"]
].to_odps_table("image_sink_table", overwrite=True).execute()

Example 2: Image preprocessing (resize, crop, and format conversion)

Resize, crop, and convert image formats to prepare training data for AI models.

df = md.read_odps_table("image_src_table")

df["img_bytes"] = df["oss_path"].url.download(
    storage_options={"role_arn": ROLE_ARN}
)
df["img_obj"] = df["img_bytes"].image.decode()

# Resize image to 224×224 (commonly used for model input)
df["img_resized"] = df["img_obj"].image.resize((224, 224))

# Crop image to specified region (left, top, right, bottom)
df["img_cropped"] = df["img_obj"].image.crop((100, 100, 500, 500))

# Convert color mode (e.g., RGBA → RGB)
df["img_rgb"] = df["img_obj"].image.convert("RGB")

Q1: How do I troubleshoot image download failures?

1. Verify the OSS path uses the internal endpoint.

2. Confirm the RAM role has OSS access permissions.

3. Check the Logview error logs for details.

Q2: How do I handle large-scale data?

Multimodal operators run in parallel automatically on the MaxFrame DPE engine — data sharding and task scheduling are handled internally, with no manual parallelism configuration required. For custom processing logic, use the MaxFrame apply_chunk operator for batch processing.