Access Unstructured Data with MaxCompute
MaxCompute provides three methods to access unstructured data: Object Table, Lance external tables, and the Blob data type.
Overview
AI workloads often involve large volumes of unstructured data (images, audio, video, and documents) stored in data lakes. MaxCompute provides lakehouse integration to access unstructured data on OSS through Object Table and Lance external tables. Object Table offers stronger metadata management, while Lance external tables emphasize ecosystem compatibility.
As a lakehouse platform, MaxCompute also supports storing unstructured binary large objects (images, audio, video, and documents) as the Blob data type in tables through its built-in open storage service.
Object Table
Created from directories that store unstructured data on OSS. MaxCompute automatically extracts file metadata (size, name, and attributes) into Object Table. Key capabilities:
-
Detect file sizes to control concurrency.
-
Filter by file name or attributes to control the read scope.
-
Use content-reading or URL functions to provide data content or location information to the SQL engine and MaxFrame engine.
Object Table integrates directly with OSS to extract, store, update, and use metadata, providing deeply optimized unstructured data management for the SQL engine and MaxFrame engine. OBJECT TABLE.
Lance external tables
The Lance format stores unstructured data, metadata, indexes, and structured data on data lakes. MaxCompute provides Lance external tables for ecosystem compatibility, mapping Lance-formatted data on data lakes to MaxCompute external tables. MaxCompute reads and writes structured data in Lance through the SQL engine and MaxFrame engine, and processes unstructured data through AI Function. Lance external tables (beta)
Blob data type
The Blob data type stores unstructured data directly within MaxCompute without depending on an external data lake (Blob overview and examples). Key features:
-
Simple read/write: Supports individual Blob fields up to 5 GB in a Delta Table;
-
Better performance : Queries that don't access the Blob column are unaffected, because Blob data is stored outside the table files;
-
SQL and MaxFrame support : When queries access the Blob column, the SQL engine and MaxFrame engine read unstructured data with high concurrency based on Blob column metadata;
-
Deep optimization : Data is stored on the MaxCompute built-in storage service, ensuring bandwidth, I/O, and security.
Data scenarios and sources
Using these three access methods, MaxCompute processes text, images, audio, and video data through AI Function on the SQL engine.
The following table lists data scenarios and sources supported by AI Function model services. Resource usage varies by model. Public models and applicable quotas .
|
Model service |
Supported models |
Data scenarios |
Data sources |
|
MaxCompute public models |
|
Text |
Object Table (URL mode) |
|
MaxCompute remote models (PAI-EAS model service) |
Supports multiple preset and custom models, including text and multimodal models |
Audio and video |
Object Table (URL mode) only Specify the data type by using the audio or video parameter. |
|
Text and images |
Object Table, Blob, and Lance external tables (Binary mode) Multimodal models process images without special parameters. Text models process text without additional parameters. |
-
URL mode: AI Function accesses Object Table data through file URLs. Suitable for large files such as audio and video.
-
Binary mode: AI Function reads binary data directly from Object Table, Blob, and Lance external tables. Suitable for text and image scenarios.