Data Lake Formation (DLF) is a fully managed data lake service that provides unified metadata, permission, and storage management and integrates with Alibaba Cloud big data compute engines.
Choose a document based on your goal
If you already know what you want to accomplish, go directly to the relevant document.
| Goal | Recommended documents |
|---|---|
| Activate and start using DLF | Set up DLF and Get started with DLF |
| Create and manage catalogs, databases, or tables | Manage catalogs and Manage tables |
| Grant data permissions to users or roles | Manage data permissions |
| Access DLF from a compute engine | Compute engine integration |
| Optimize lake table files and storage | Storage optimization |
| Manage and retrieve multimodal data | Multimodal data lake |
| Manage DLF by using APIs or the CLI | API user guide and CLI reference |
If you are new to DLF, set up the service first and then follow the getting-started guide to complete your first task.
Core capabilities
Unified metadata and storage
DLF provides a single set of lakehouse metadata and storage shared across all integrated compute engines. Data flows between products without manual synchronization or schema duplication.
Unified permission management
Define access control once at the Catalog, database, or table level. Every integrated service enforces the same permissions, eliminating separate permission configurations per engine.
Storage optimization
DLF automates lakehouse table maintenance through configurable strategies:
| Strategy | Effect |
|---|---|
| File compaction | Merges small files to improve scan performance |
| Expired snapshot cleanup | Removes outdated snapshots to reclaim storage |
| Expired partition cleanup | Deletes data from expired partitions |
| Orphaned file cleanup | Removes files no longer referenced by any table version |
Multimodal data management and retrieval
DLF manages multimodal data that contains images, vectors, and documents and supports image search, full-text search, and visual exploration.
Compute engine integration
Alibaba Cloud stream and batch compute engines can access and use data managed in DLF.
Architecture
DLF's architecture has three layers:
| Layer | Capabilities |
|---|---|
| Metadata management | Manage catalogs, databases, tables, and their metadata. |
| Permission management | Manage data permissions at the catalog, database, and table levels. |
| Storage optimization | Manage file compaction, expired snapshot cleanup, expired partition cleanup, and orphaned file cleanup strategies. |
Use cases
Data lakehouse
DLF provides unified metadata, storage, and permission management for historical, real-time, and multimodal data and allows compute engines to access governed data.
Traditional big data
DLF supports offline analytics, real-time analytics, machine learning, and log analytics. Unified metadata and storage management simplify data lake construction and governance.
What to do next
| Goal | Next step |
|---|---|
| Understand costs | See Billing. |
| Start using DLF | See Get started with DLF. |
| Try multimodal retrieval | See Get started with DLF multimodal retrieval. |
| Follow an end-to-end implementation | See Build a streaming lakehouse by using DLF and Flink. |