Data Lake Formation (DLF) provides table management APIs compatible with Apache Paimon's REST Catalog. Its fully Paimon-compatible file storage structure allows any Paimon-aware engine or application to easily create, update, query, and delete tables managed by DLF.
The main data hierarchy in a catalog is as follows:
Catalog: The top-level logical entity for metadata organization. A catalog structures metadata resources hierarchically, enabling isolation and permission control across services and users. It also manages data lake storage and lake table operations.
Database: A logical grouping of metadata within a catalog, offering finer-grained organization and access control than a catalog.
Tables: DLF supports diverse table types for unified management and seamless compatibility with various compute engines and formats. Data encryption at rest is available upon request via ticket submission.
View: Persisted in DLF, views support multiple SQL dialects, allowing configuration for different compute engines.
Function: Persisted in DLF, functions currently support Flink JARs (Java and Python) and Java Lambda functions executed on the Spark engine.