Hightopo and ApsaraDB for Lindorm (Lindorm) cooperate to develop a hyper-converged storage architecture for the industrial IoT (IIoT). This storage architecture is suitable for the industrial, manufacturing, building, and aerospace industries.
- One-stop storage system for hundreds of petabytes of heterogeneous monitoring data from multiple sources
- Tens of millions of throughput per second that allows highly concurrent writes of data for monitoring metrics
- Collaboration of multiple engines for multi-model data retrieval to support complex data virtualization scenarios
The data in the IIoT scenarios requires highly concurrent writes and real-time data access. Lindorm integrates the capabilities of multiple engines, such as a time series engine, search engine, and wide table engine, as a solution. Lindorm provides an optimal method to store and analyze monitoring data that features low value density, high throughput, and high timeliness at a low cost. It greatly reduces the data storage cost and the O&M cost of storage systems.
Hightopo was founded in 2013. Its headquarters is in Xiamen and has branches in other cities, such as Beijing, Shanghai, Tianjin, Dalian, and Qingdao. The company focuses on visual monitoring and visual O&M for the IIoT. The company provides customers with end-to-end visualization management and support, from consultation, design, and implementation to after-sales services. Hightopo is dedicated to the component technologies of web-based two-dimensional (2D) and three-dimensional (3D) graphic interfaces. It has launched a proprietary software application named HT for Web and continuously strives to make HT for Web one of the top software applications worldwide. HT for Web has been applied in various industries, such as telecommunications, electricity, hydropower, transportation, petrochemical, manufacturing, medical, and industrial control industries.
Business requirements and challenges
Due to the rapid development of AI and Internet technologies such as 5G, cloud computing, and edge computing, a sharply increasing number of software sensors and hardware sensors are used in the IIoT. This results in rapid increases in the amount of IIoT data to be collected and the number of data types. These changes make it more difficult to store and query the IIoT data. Before Hightopo switched to Lindorm, Hightopo used a single self-managed engine, such as Elasticsearch, OpenTSDB, Prometheus, or another time series engine. However, this solution is no longer applicable due to the complex storage technologies and increased O&M costs caused by the diversified types of data to collect. Hightopo needs a new-generation cloud native storage solution that provides multi-model and data retrieval capabilities. A well-known IT consulting company predicts that the IIoT market size may reach USD 3.7 trillion by 2025. However, statistics show that less than 30% of IIoT vendors profit from the IIoT market. Most IIoT companies cannot make profits because of high technology costs. New technologies create opportunities for industry upgrades and also bring challenges such as more complex system architectures and higher requirements for performance and stability. These challenges limit the implementation of IIoT systems. To handle these challenges, IIoT companies need to seek help from professional technology service providers who can address the difficulties in building end-to-end data processing solutions. These difficulties include data collection, transmission, storage, analysis, and visualization.
Hightopo is dedicated to handling the technical difficulties in the last phase of the data processing procedure in the IIoT: data visualization. This company provides a visualization solution for the monitoring systems used in the IIoT. In this solution, HT for Web is suitable for the UI display of real-time monitoring systems. After the solution is applied, topological graphs, charts, and other highly interactive applications are easy to create, deploy, and customize. This solution is widely used in telecommunication networks to manage topologies and devices. It is also widely used in the electricity industry, gas industry, and other industrial automation fields such as human-machine interfaces (HMIs) and supervisory control and data acquisition (SCADA). In some scenarios such as production lines, situation awareness in smart transportation, and wind farms, large volumes of data need to be collected in real time. To collect, store, index, and aggregate data in these scenarios, Hightopo previously used Elasticsearch to store data collected from sensors, Prometheus to store data collected from third-party systems, and HBase to store data collected from the end devices of users. The collected data included time series metrics, log data, user experience data, and network traffic data. Due to an increase in the amount of data, data virtualization scenarios became complex, the storage cost and O&M cost sharply increased, data retrieval became complex, and the user experience was greatly compromised.
To address storage issues, Hightopo optimizes its data storage architecture by using Lindorm. Lindorm uses a multi-model hyper-converged storage architecture that allows Hightopo to use a single Lindorm database to integrate data of multiple models. Hightopo uses Lindorm to collect the full monitoring data. This simplifies the storage architecture and reduces the O&M cost. Figure 2 compares the storage architecture that Hightopo previously used and the data storage architecture that is optimized by using Lindorm. Lindorm provides a data compression feature and a feature that can be used to optimize data compression. After the two features are enabled, the costs of storing large amounts of monitoring data that has low value density are significantly reduced. A single Lindorm instance on the cloud can use multiple engines provided by Lindorm, such as the wide table engine, search engine, and time series engine. This way, Lindorm can collect data from increasingly complex and diverse data sources such as end devices, edge devices, sensors, and third-party systems and report heterogeneous data. Figure 3 shows the architecture of Lindorm. To facilitate the interaction of data from different engines, Lindorm can use Alibaba Cloud Data Transmission Service (DTS), Alibaba Cloud Data Management (DMS), Apache NiFi, Apache Sqoop, or other third-party open source data transfer services or extract, transform, load (ETL) services. Lindorm selects suitable data based on the business requirements of specific scenarios. Lindorm provides an SDK and a RESTful API to simplify data access from upper-layer data visualization and analysis systems. Lindorm is compatible with the native APIs provided by OpenTSDB, Prometheus, and HBase. This ensures that Lindorm can seamlessly connect to major services and tools in the ecosystem and reduces the cost of integrating and deploying HT for Web.
- Real-time monitoring dashboards
- Situation awareness and risk monitoring
- Defect detection by using device monitoring data
- Tracing and analysis of all faults
- AI-powered anomaly detection
- This multi-model storage structure can be used to store large amounts of heterogeneous data. This reduces the storage cost and data aggregation cost in the IIoT.
- The storage for monitoring data has high performance and supports high throughput. This improves the timeliness of data displayed on visual monitoring dashboards.
- The 99.95% reliability provided by Lindorm ensures stable and smooth running of the business.
- Lindorm can be connected anywhere. This simplifies the network configuration and management.
- Lindorm is an out-of-box service. It frees the company from O&M. This reduces the system O&M costs.
- Lindorm is compatible with the APIs provided by OpenTSDB and Prometheus. This ensures that Lindorm can seamlessly connect to major services and tools in the ecosystem.
Hightopo is used together with Lindorm to help customers collect, store, and retrieve the data from user terminals and sensors. These customers include a company in the Ubiquitous Electric Internet of Things (UEIoT) industry and a company active in the smart building field. After the solution is applied in the customers' monitoring systems, the concurrency reaches up to 1 million transactions per second (TPS) and up to 400 thousand time series in the time series can be stored. The solution also reduces the storage and system maintenance cost by 60%.