This topic provides an overview for Lindorm.
Overview
Lindorm is a cloud-native hyper-converged database service that is developed and optimized to store and process multimodal data in various scenarios such as IoT, Internet, and Internet of Vehicles (IoV). Lindorm provides unified capabilities for database access and integrated processing capabilities for multiple types of data, such as wide tables, time series, files, objects, streams, and spatial data. Lindorm is compatible with the standard APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, Amazon Simple Storage Service (Amazon S3), OpenTSDB, Hadoop Distributed File System (HDFS), Apache Solr, and Apache Kafka. It can also be seamlessly integrated with third-party ecosystem tools. Lindorm is suitable for scenarios such as log data processing, monitoring, bill data processing, advertising, social networking, traveling, and risk management. It also provides strong support for the core business of Alibaba Group.
Lindorm uses a cloud-native architecture that decouples computing and storage resources and integrates multiple engines with shared underlying storage for multimodal data. Characterized by its benefits such as elasticity, cost-effectiveness, ease of use, high compatibility, and high stability, Lindorm allows you to store and analyze various types of data, such as metadata, logs, bills, tags, messages, reports, dimension tables, result tables, feeds, user personas, device data, monitoring data, sensor data, small files, and small pictures. Lindorm provides the following core capabilities:
Core capability | Overview |
Integration of multiple data models | Lindorm supports multiple types of data models, such as wide tables, time series, objects, files, queues, and spatial data. Data can be transferred and synchronized between models. Lindorm provides unified and integrated capabilities and services, including data access, storage, retrieval, computing, and analysis. This helps make application development more agile, flexible, and efficient. |
Cost-effectiveness | Lindorm can handle tens of millions of concurrent requests and respond to requests at a latency of a few milliseconds. Lindorm automatically stores hot and cold data by using separate storage media and adaptively compress adaptive compresses data based on data characteristics for high cost-effectiveness. |
Cloud native elasticity | Lindorm supports the individual scaling of computing and storage resources. |
Compatibility | Lindorm is compatible with the standard APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, Amazon S3, Apache Phoenix, OpenTSDB, HDFS, Apache Solr, and Apache Kafka. It can be seamlessly integrated with Hadoop, Spark, Flink, and Kafka systems, and provides easy-to-use features that allow you to transfer, process, and subscribe to data. |
Multimodal capabilities
Lindorm supports multiple types of data models, including wide tables, time series, objects, files, queues, and spatial data. It supports standard SQL statements and the APIs of multiple open source systems. Data can be transferred and synchronized between models. This helps make application development more agile, flexible, and efficient. The following table describes the core multimodal capabilities provided different Lindorm engines.
Engine | Core capability |
Wide table engine (LindormTable) | LindormTable boasts various features such as global secondary indexes, multi-dimensional queries, dynamic columns, and Time to Live (TTL) to manage wide tables and objects. This makes it suitable for the storage and management of metadata, orders, bills, user personas, social networking information, feeds, and logs. LindormTable is compatible with the open APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, and Amazon S3. LindormTable can handle tens of millions of concurrent requests and store hundreds of petabytes of data. It also provides the hot and cold data separation feature. Compared with the performance of open source Apache HBase, the throughput is increased by 2 to 6 times, the percentile 99%(P99) latency is decreased by 90%, the mean time to repair (MTTR) is decreased by 90%, the data compression ratio is increased by 100%, and the comprehensive storage cost is decreased by 50%. |
Time series engine (LindormTSDB) | LindormTSDB is used to manage time series data, such as the measurement data, monitoring data, and operational data of devices in the industrial sector and scenarios such as IoT and monitoring. It allows you to execute SQL statements to manage, write, and query time series data. LindormTSDB uses a dedicated compression algorithm to compress time series data at a compression ratio up to 15:1. LindormTSDB supports multi-dimensional queries and aggregate computing of large amounts of data. It also supports downsampling and continuous queries. |
Search engine (LindormSearch) | Developed based on core technologies such as columnar storage and inverted indexing, LindormSearch is used to accelerate the retrieval and analysis of multimodal data. LindormSearch boasts various capabilities such as full-text retrieval, aggregate computing, and complex multi-dimensional queries and is suitable for scenarios such as the queries of logs, bills, and user personas. It is also compatible with the standard APIs of open source software and services such as SQL and Apache Solr. |
Compute engine (LDPS). | Lindorm Distributed Processing System (LDPS) is integrated with LindormDFS to provide distributed computing services based on a cloud-native architecture to meet computing requirements in various scenarios, such as data production, interactive analytics, machine learning, and graph computing. LDPS is compatible with the standard Apache Spark API. |
Streaming engine | The Lindorm streaming engine is used to store and process streaming data. It provides the storage and lightweight computing capabilities that help you store streaming data and build applications to process the data. |
Vector engine | The Lindorm vector engine is used to store, index, and retrieve large amounts of vector data. It supports multiple indexing algorithms, distance functions, and a variety of integrated data retrieval methods. The Lindorm vector engine provides the capability to perform full-text and vector retrievals in an integrated manner, which is required in RAG to improve the accuracy of large models. Therefore, the Lindorm vector engine is suitable for AI-related services such as recommendation, NLP, and intelligent Q&A. |