This topic describes the overview of Lindorm.
Overview
Lindorm is a cloud native multi-model hyper-converged database service that is developed and optimized for IoT, Internet, and Internet of Vehicles (IoV). Lindorm provides unified capabilities for database access and integrated processing capabilities for multiple types of data, such as wide tables, time series, files, objects, streams, and spaces. Lindorm is compatible with the standard APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, Amazon Simple Storage Service (Amazon S3), Apache Phoenix, OpenTSDB, Hadoop Distributed File System (HDFS), Apache Solr, and Apache Kafka. It can also be seamlessly integrated with third-party ecosystem tools. Lindorm is suitable for scenarios such as log data processing, monitoring, bill data processing, advertising, social networking, traveling, and risk management. It also provides strong support for the core business of Alibaba Group.
Lindorm uses a cloud native multi-model architecture in which computing is decoupled from storage. Lindorm provides benefits such as elasticity, cost-effectiveness, ease of use, high compatibility, and high stability. It allows you to store and analyze data such as metadata, logs, bills, tags, messages, reports, dimension tables, result tables, feeds, user personas, device data, monitoring data, sensor data, small files, and small pictures. Lindorm provides the following core capabilities:
Multi-model integration: Lindorm supports multiple types of data models, such as wide tables, time series, objects, files, queues, and spaces. Data can be transferred and synchronized between models. Lindorm provides unified and integrated capabilities and services, including data access, storage, retrieval, computing, and analysis. This helps make application development more agile, flexible, and efficient.
High cost-effectiveness: Lindorm can handle tens of millions of concurrent requests and responds at a latency of a few milliseconds. Lindorm supports multi-level media for data storage and provides the automatic cold and hot data separation feature and the adaptive compression feature. This helps reduce the storage cost.
Cloud native elasticity: Lindorm provides separate auto scaling for computing resources and storage resources.
High compatibility: Lindorm is compatible with the standard APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, Amazon S3, Apache Phoenix, OpenTSDB, HDFS, Apache Solr, and Apache Kafka. It can be seamlessly integrated with Hadoop, Spark, Flink, and Kafka systems, and provides easy-to-use features that allow you to transfer, process, and subscribe to data.
Multi-model capabilities
Lindorm supports multiple types of data models, including wide tables, time series, objects, files, queues, and spaces. It supports standard SQL statements and supports the APIs of multiple open source systems. Data can be transferred and synchronized between models. This helps make application development more agile, flexible, and efficient. The core multi-model capabilities are provided by the following data engines:
Wide table engine
The wide table engine is used to provide services to manage wide table data and object data. It provides global secondary indexes, multi-dimensional queries, dynamic columns, and the Time to Live (TTL) feature. This engine is suitable for scenarios such as the storage and management of metadata, orders, bills, user personas, social networking information, feeds, and logs. The wide table engine is compatible with the open APIs of multiple open source software and services, such as SQL, Apache HBase, Apache Cassandra, and Amazon S3.
The wide table engine can handle tens of millions of concurrent requests and store hundreds of petabytes of data. It also provides the hot and cold data separation feature. Compared with the performance of open source Apache HBase, the throughput is increased by 2 to 6 times, the percentile 99% (P99) latency is decreased by 90%, the mean time to repair (MTTR) is decreased by 90%, the data compression ratio is increased by 100%, and the comprehensive storage cost is decreased by 50%.
Time series engine
The time series engine is used to provide services to manage time series data such as measurement data, monitoring data, and operational data of devices in the industrial sector or scenarios such as IoT and monitoring. It allows you to execute SQL statements to manage, write, and query time series data. The time series engine uses a dedicated compression algorithm that is provided for time series data. The data compression ratio can reach up to 15:1. The time series engine supports multi-dimensional queries and aggregate computing of large amounts of data. It also supports downsampling and continuous queries.
Search engine
The search engine is used to accelerate the retrieval and analysis of multi-model data. It is developed based on core technologies such as columnar storage and inverted indexing to provide capabilities such as full-text indexes, aggregate computing, and complex multi-dimensional queries. The search engine is suitable for scenarios such as the queries of logs, bills, and user personas. It is compatible with the standard APIs of open source software and services such as SQL and Apache Solr.
File engine
The file engine is used to provide services to manage data directories and data files. It supports access to the underlying storage that is shared by the file engine, wide table engine, time series engine, and search engine. This way, the underlying data files can be imported, exported, computed, and analyzed in a more efficient manner. The file engine is compatible with the standard HDFS API.
Compute engine
The compute engine is integrated with the Lindorm storage engine. The compute engine provides distributed computing services based on a cloud native architecture to meet your computing requirements in various scenarios, such as data production, interactive analytics, machine learning, and graph computing. The compute engine is compatible with the standard Apache Spark API.
Stream computing engine
The Lindorm stream computing engine is used to store and process streaming data. It provides lightweight computing capabilities. You can use the stream computing engine to store streaming data to Lindorm to meet the requirements for the processing and application of streaming data.