Data Lake Formation (DLF) is a fully managed data lake service that provides unified metadata, permission, and storage management and integrates with Alibaba Cloud big data compute engines.
Core capabilities
Unified metadata and storage
DLF provides a single set of lakehouse metadata and storage shared across all integrated compute engines. Data flows between products without manual synchronization or schema duplication.
Unified permission management
Define access control once at the Catalog, database, or table level. Every integrated service enforces the same permissions, eliminating separate permission configurations per engine.
Storage optimization
DLF automates lakehouse table maintenance through configurable strategies:
| Strategy | Effect |
|---|---|
| File compaction | Merges small files to improve scan performance |
| Expired snapshot cleanup | Removes outdated snapshots to reclaim storage |
| Expired partition cleanup | Deletes data from expired partitions |
| Orphaned file cleanup | Removes files no longer referenced by any table version |
Multimodal data management and retrieval
DLF manages multimodal data that contains images, vectors, and documents and supports image search, full-text search, and visual exploration.
Compute engine integration
Alibaba Cloud stream and batch compute engines can access and use data managed in DLF.
Architecture
DLF's architecture has three layers:
| Layer | Capabilities |
|---|---|
| Metadata management | Manage catalogs, databases, tables, and their metadata. |
| Permission management | Manage data permissions at the catalog, database, and table levels. |
| Storage optimization | Manage file compaction, expired snapshot cleanup, expired partition cleanup, and orphaned file cleanup strategies. |
Benefits
| Benefit | Description |
|---|---|
| Fully managed | Available out-of-the-box with no infrastructure to provision or maintain. Supports the full data lifecycle with unified Paimon metadata and storage management. |
| Enterprise-level security | Dual control over API and data permissions across multiple abstraction levels keeps data secure and compliant. |
| Flexible optimization strategies | Configurable file compaction and data cleanup strategies improve access performance and lower storage costs. |
| Rich ecosystem | Built on deep Paimon integration, DLF connects your data lakehouse to Alibaba Cloud compute engines and AI products through a single managed service. |
Use cases
Data lakehouse
DLF provides unified metadata, storage, and permission management for historical, real-time, and multimodal data and allows compute engines to access governed data.
Traditional big data
DLF supports offline analytics, real-time analytics, machine learning, and log analytics. Unified metadata and storage management simplify data lake construction and governance.