This topic provides an overview for Lindorm.
What is Lindorm
Lindorm is a cloud-native, multi-model, hyper-converged database designed and optimized for scenarios like the Internet of Things (IoT), internet applications, and the Internet of Vehicles (IoV). It provides unified access and integrated processing for various data types, such as wide table, time series, text, object, stream, and spatial data. Lindorm is compatible with standard interfaces such as SQL, HBase/S3, TSDB, HDFS, Solr, and Kafka, and it seamlessly integrates with third-party ecosystem tools. It is suitable for scenarios such as logging, monitoring, billing, advertising, social networking, mobility, and risk control. Lindorm is also one of the databases that power Alibaba's core businesses.
Why choose Lindorm
Business background
Traditional data storage architectures often use separate databases for different data types, such as relational data, time series data, and object storage. This leads to significant technological fragmentation: a complex stack of components, difficult technology choices, and long data synchronization pipelines. This approach demands more specialized expertise from technical staff, lengthens business deployment cycles, increases failure rates and operational costs, and hinders architectural evolution. For example, achieving high availability across availability zones or global replication often requires separate modifications for each component. The adoption of 5G and IoT technologies further amplifies the core conflict between diverse data needs and increasingly complex storage architectures.
To address these issues, Alibaba Cloud developed Lindorm to provide unified storage, querying, and analysis for multi-model data. As shown in the following figure, Lindorm's architecture greatly simplifies the data storage stack compared to traditional solutions, which improves system stability and reduces deployment costs.
Core capabilities
Lindorm is elastic, cost-effective, easy to use, open, and stable. It is ideal for storing and analyzing various data types, such as metadata, logs, bills, tags, messages, reports, dimension tables, result tables, feed streams, user profiles, device data, monitoring data, sensor data, small files, and small images. The core capabilities of Lindorm are:
Core capability | Description |
Multi-model hyper-convergence | Supports multiple data models, such as wide table, time series, object, text, queue, and spatial. Data is interoperable across these models. Lindorm provides integrated capabilities and services for data ingestion, storage, retrieval, computing, and analysis. This lets you build applications faster with more flexibility and efficiency. |
Cost-effectiveness | Delivers tens of millions of concurrent requests and millisecond-level latency. It significantly reduces storage costs by using tiered storage, intelligent hot and cold data separation, and adaptive compression based on data characteristics. |
Cloud-native elasticity | Supports independent elastic scaling of compute and storage resources. |
Open compatibility | Compatible with multiple standard interfaces, such as SQL, HBase/S3, TSDB, HDFS, Solr, and Kafka. Lindorm seamlessly integrates with systems such as Hadoop, Spark, Flink, and Kafka, enabling straightforward data exchange, processing, and subscription. |
For more information, see Features and Advantages.
Product architecture
Lindorm uses an innovative cloud-native architecture with compute-storage separation and shared multi-model integration to support the demands of the cloud era for resource decoupling and elastic scaling. Lindorm is built on LindormDFS, a cloud-native distributed file system that acts as its unified storage foundation. This foundation supports various specialized data engines, including the wide table engine, time series engine, search engine, vector engine, and columnar engine. On top of these engines, Lindorm provides a unified SQL interface for cross-model federated queries and also supports open-source standard interfaces such as HBase, OpenTSDB, Spark, and HDFS to ensure seamless migration for existing workloads. Additionally, Lindorm Tunnel Service (LTS) handles real-time data flow and change data capture between engines, enabling data migration, real-time subscription, data lake offloading, data warehouse write-back, unitized multi-active architectures, and backup and recovery.
Multimodal capabilities
Lindorm supports multiple types of data models, including wide tables, time series, objects, files, queues, and spatial data. It supports standard SQL statements and the APIs of multiple open source systems. Data can be transferred and synchronized between models. This helps make application development more agile, flexible, and efficient. The following table describes the core multimodal capabilities provided different Lindorm engines.
Engine | Core capability |
Wide table engine (LindormTable) | Manages and serves wide table and object data. It provides features such as global secondary indexes, multi-dimensional retrieval, dynamic columns, and Time-to-Live (TTL). This engine is suitable for scenarios such as metadata, orders, bills, user profiles, social feeds, and logs. It is compatible with standard interfaces such as SQL, HBase, and S3. LindormTable can handle tens of millions of concurrent requests and store hundreds of petabytes of data. It also provides the hot and cold data separation feature. Compared with the performance of open source Apache HBase, the throughput is increased by 2 to 6 times, the percentile 99%(P99) latency is decreased by 90%, the mean time to repair (MTTR) is decreased by 90%, the data compression ratio is increased by 100%, and the comprehensive storage cost is decreased by 50%. |
Time series engine (LindormTSDB) | Manages and serves time series data. It is designed for measurement data, monitoring data, and equipment operational data in industrial, IoT, and monitoring scenarios. It provides SQL-based management, write, and query capabilities. Its specialized compression algorithm for time series data achieves a compression ratio of up to 10:1. The engine supports multi-dimensional queries and aggregate computing on massive datasets, provides native PromQL query capabilities, and supports pre-downsampling and continuous queries for time series data. |
Search engine (LindormSearch) | Developed based on core technologies such as columnar storage and inverted indexing, LindormSearch is used to accelerate the retrieval and analysis of multimodal data. LindormSearch boasts various capabilities such as full-text retrieval, aggregate computing, and complex multi-dimensional queries and is suitable for scenarios such as the queries of logs, bills, and user personas. It is also compatible with the standard APIs of open source software and services such as SQL and Apache Solr. |
Compute engine (LDPS). | Lindorm Distributed Processing System (LDPS) is integrated with LindormDFS to provide distributed computing services based on a cloud-native architecture to meet computing requirements in various scenarios, such as data production, interactive analytics, machine learning, and graph computing. LDPS is compatible with the standard Apache Spark API. You are the owner of the LDPS resources in your Lindorm instances. |
Vector engine | The Lindorm vector engine is used to store, index, and retrieve large amounts of vector data. It supports multiple indexing algorithms, distance functions, and a variety of integrated data retrieval methods. The Lindorm vector engine provides the capability to perform full-text and vector retrievals in an integrated manner, which is required in RAG to improve the accuracy of large models. Therefore, the Lindorm vector engine is suitable for AI-related services such as recommendation, NLP, and intelligent Q&A. |
Columnar engine (L-Column) | The Lindorm columnar engine is a high-performance, low-cost, stable, and reliable online columnar database engine service. It targets IoT, IoV, and logging scenarios, and provides efficient read/write operations, high-compression storage, and high-performance online analytics. |
AI Engine (Lindorm AI) | The Lindorm AI engine provides a one-stop integrated AI inference capability that runs on your resources. You can use Lindorm SQL to flexibly import and deploy pre-trained models for intelligent analysis and processing of massive multi-model data. |