Lindorm Tunnel Service (LTS) is a data ecosystem service that is customized based on the characteristics of business scenarios in which Lindorm is used. LTS provides easy-to-use capabilities, including data exchange, processing, and change tracking. You can use these capabilities to migrate data, track real-time data changes, dump data to data lakes, and synchronize data from data warehouses to Lindorm databases. You can also use these capabilities to back up and restore data, and implement multi-active redundancy based on units. This way, LTS provides an all-in-one data ecosystem service for Lindorm.
Core capabilities
Cloud native distributed system: LTS is a distributed system that is deployed based on Elastic Compute Service (ECS). LTS features excellent horizontal scalability and allows you to configure resources based on your business requirements.
Ease of use: LTS allows you to configure data migration, import, change tracking, and archiving tasks. For example, to create a data migration task, you need only to specify the source, destination, and columns that you want to synchronize. LTS automatically replicates schemas, full data, or incremental data based on your settings.
High security and reliability: LTS minimizes the impact on the source and destination systems and minimizes the risks of potential failures due to incompatibility. Before a task is started, LTS prechecks the network connectivity and security. While the task is running, LTS monitors the synchronization latency and the storage usage of the destination cluster in real time. LTS also implements throttling and reports alerts based on the monitoring data. After the task is complete, LTS verifies data.
Cost-efficiency: LTS is an optimized service based on open source systems, such as Apache HBase, Apache Phoenix, and Apache Cassandra. LTS allows you to process data at the physical file level. This is 10 times more efficient than traditional data replication. LTS also provides optimized CPU, cache, memory, and network I/O capabilities. This enables LTS to provide cost-effective tunnels and helps reduce your costs of data transfer and processing.
Features
Feature | Scenario | References |
Data migration between HBase and LindormTable | Seamless data migration between existing clusters and new clusters, cluster upgrades, online and offline workload decoupling, primary/secondary disaster recovery, and active geo-redundancy. | |
RDS -> Lindorm Important This feature is no longer available for LTS instances that are purchased after March 10, 2023. If your LTS instance is purchased before March 10, 2023, you can still use this feature. | Online and offline workload decoupling and historical data archiving. | |
MaxCompute/Hive -> Lindorm | Offline query acceleration, and the transmission of details and metrics from data warehouses to Lindorm for online queries. | For more information, contact the technical support. |
Data export from Lindorm to MaxCompute (previously known as Open Data Processing Service (ODPS)) Important This feature is no longer available for LTS instances that are purchased after June 16, 2023. If your LTS instance is purchased before June 16, 2023, you can still use this feature. | The export of historical data and incremental data. | Export full data to MaxCompute and Archive incremental data to MaxCompute |
Subscription to real-time data in LogHub Important This feature is no longer available for LTS instances that are purchased after June 16, 2023. If your LTS instance is purchased before June 16, 2023, you can still use this feature. | The subscription to real-time data from LogHub and the consumption of the data in Lindorm | |
Change tracking | The subscription to real-time incremental data in Lindorm. |
Log lifecycle management
If log data is not consumed after you enable the log subscription feature, the log data is retained for 48 hours by default. After the period expires, the subscription is automatically canceled and the retained data is automatically deleted.
Log data may fail to be consumed if your LTS cluster is released while your task is still running or if your synchronization task is suspended.
You can enable the log subscription feature for the following types of tasks in Lindorm: incremental synchronization, data archiving, data backup, and data subscription.
Scenarios
Cluster migration
Usage scope
Data migration from HBase to Lindorm.
The switchover of cluster networks. For example, the network type is changed from the classic network to a virtual private cloud (VPC).
Data center migration across regions.
Workload decoupling.
Benefits
Data can be migrated without service interruption. LTS can migrate historical data and synchronize real-time incremental data in one task.
When data is being migrated, LTS does not interact with the source HBase or Lindorm cluster. LTS reads data only from the HDFS of the source cluster. This minimizes the impact on the online business that runs on the source cluster.
In most cases, compared with data migration at the API layer, data replication at the file layer can help you reduce more than 50% of the data usage.
LTS is efficient. Each node can migrate data at a rate of up to 100 MB/s. You can add nodes for horizontal scaling to migrate terabytes or even petabytes of data.
LTS provides stable services by retrying failed tasks, monitoring the synchronization rates and progress of tasks in real time, and reporting alerts when tasks fail.
LTS ensures data accuracy by verifying the synchronized data.
Automatic schema synchronization is supported to ensure consistent partitions.
Online and offline workload decoupling
LTS allows you to synchronize online business data in real time to HDFS or OSS storage. LTS can work with components of big data services, such as Spark and MapReduce, to analyze data. This ensures that online business queries are not affected.
Primary/secondary disaster recovery
LTS supports two-way data synchronization between an active cluster and a standby cluster. When the active cluster fails, you can switch to the standby cluster to reduce the impact on your workloads. After the active cluster recovers, you can use LTS to synchronize the incremental data from the standby cluster to the active cluster.
Historical data storage in ApsaraDB RDS databases
In scenarios where historical data, such as transaction orders, is stored, performance bottlenecks may occur in ApsaraDB RDS databases due to the ever-increasing data size. Periodic data archiving or sharding is complicated and causes high costs. LTS allows you to synchronize data from ApsaraDB RDS to LindormTable in real time to separate hot data from cold data. LindormTable supports automatic horizontal scaling, high-concurrency queries, multi-dimensional indexing, and lightweight analysis. Lindorm Streams allows you to track data changes in sequence. LTS also allows you to synchronize data from LindormTable to other analytics systems for complex data analysis.