MaxCompute offers functions to process unstructured data, letting you access data and metadata stored in a data warehouse or data lake.
Built-in functions for unstructured data
MaxCompute supports the Object Table feature, which allows the data warehouse computing engine to access unstructured data and its metadata stored in OSS. The system also provides corresponding functions for processing OSS file data. For more information, see Object Table definition.
Function | Description |
Reads all or part of an object's content and returns it as binary data. | |
Generates a pre-signed URL that you can use to directly read from or write to OSS files over HTTP. | |
Generates a blob reference to an OSS object. The reference contains information such as the storage location, byte range, and an optional Content-Type. |
Custom UDFs for unstructured data
The following custom UDF examples demonstrate how to use Object Table to access unstructured data and its metadata in a data lake.
Type | Function | Description |
Metadata processing (Java) | Gets a file's metadata. | |
Extracts the format of a file, such as | ||
Detects the top-level MIME media type of a file based on the IANA registry. | ||
Returns the full MIME content type (Content-Type) of a file. | ||
Extracts the filename from a file path. | ||
Content processing (Python) | Extracts the content from a PDF file as text. |
Archive file functions
MaxCompute provides two new built-in scalar functions to parse binary data from archive files directly in SQL. You can use them to read the metadata and content of files within various archive formats, such as zip, jar, tar, tar.gz, tar.bz2, tar.xz, and tar.zst, without decompressing the archive file to disk.
Function | Description |
Lists the metadata of all entries in an archive file. | |
Extracts a mapping from filenames to their content from an archive file. |