Lindorm Tunnel Service (LTS) is a data ecosystem service that is customized based on the characteristics of business scenarios in which Lindorm is used. LTS provides easy-to-use capabilities, including data exchange, processing, and change tracking. You can use these capabilities to migrate data, track real-time data changes, dump data to data lakes, and synchronize data from data warehouses to Lindorm databases. You can also use these capabilities to back up and restore data, and implement multi-active redundancy based on units. This way, LTS provides an all-in-one data ecosystem service for Lindorm.
Core capabilities
Cloud-native and distributed: LTS is deployed on ECS and supports horizontal scaling and on-demand resource configuration.
Easy to use: You can configure tasks such as data migration, import, change tracking, and archiving with a single click. For data migration, you can simply select the source, destination, and columns to synchronize. LTS automatically completes schema replication and full and incremental data synchronization.
Secure and reliable: LTS minimizes the impact on source and destination systems. Before a task starts, LTS automatically checks network connectivity and security. During the task, it monitors synchronization latency and the storage usage of the destination cluster in real time, and provides throttling and alerts. After the task finishes, LTS provides data verification.
Cost-effective: LTS is highly optimized for open source systems such as HBase and Cassandra. It improves efficiency by 10 times by processing data at the physical file level. Comprehensive optimization of CPU, cache, memory, and network I/O reduces data transfer costs.
Features
Feature | Scenario | References |
Data migration between HBase and LindormTable | Seamless data migration between existing clusters and new clusters, cluster upgrades, online and offline workload decoupling, primary/secondary disaster recovery, and active geo-redundancy. | |
RDS -> Lindorm Important This feature is no longer available for LTS instances that are purchased after March 10, 2023. If your LTS instance is purchased before March 10, 2023, you can still use this feature. | Online and offline workload decoupling and historical data archiving. | Import full data and incremental data from ApsaraDB RDS Note After this feature is discontinued, you can use DMS to synchronize data. |
MaxCompute/Hive -> Lindorm | Offline query acceleration, and the transmission of details and metrics from data warehouses to Lindorm for online queries. | For more information, contact the technical support. |
Data export from Lindorm to MaxCompute (previously known as Open Data Processing Service (ODPS)) Important This feature is no longer available for LTS instances that are purchased after June 16, 2023. If your LTS instance is purchased before June 16, 2023, you can still use this feature. | The export of historical data and incremental data. | Export full data to MaxCompute and Archive incremental data to MaxCompute |
Subscription to real-time data in LogHub Important This feature is no longer available for LTS instances that are purchased after June 16, 2023. If your LTS instance is purchased before June 16, 2023, you can still use this feature. | The subscription to real-time data from LogHub and the consumption of the data in Lindorm | |
Lindorm change tracking | The subscription to real-time incremental data in Lindorm. |
Scenarios
Cluster migration
Usage scope
Data migration from HBase to Lindorm.
The switchover of cluster networks. For example, the network type is changed from the classic network to a virtual private cloud (VPC).
Data center migration across regions.
Workload decoupling.
Features
Data can be migrated without service interruption. LTS can migrate historical data and synchronize real-time incremental data in one task.
When data is being migrated, LTS does not interact with the source HBase or Lindorm cluster. LTS reads data only from the HDFS of the source cluster. This minimizes the impact on the online business that runs on the source cluster.
In most cases, compared with data migration at the API layer, data replication at the file layer can help you reduce more than 50% of the data usage.
LTS is efficient. Each node can migrate data at a rate of up to 100 MB/s. You can add nodes for horizontal scaling to migrate terabytes or even petabytes of data.
LTS provides stable services by retrying failed tasks, monitoring the synchronization rates and progress of tasks in real time, and reporting alerts when tasks fail.
LTS ensures data accuracy by verifying the synchronized data.
Automatic schema synchronization is supported to ensure consistent partitions.
Online and offline workload decoupling
LTS lets you synchronize online business data in real time to HDFS or OSS storage. LTS can work with components of big data services, such as Spark and MapReduce, to analyze data. This ensures that online business queries are not affected.
Primary/secondary disaster recovery
LTS supports two-way data synchronization between an active cluster and a standby cluster. When the active cluster fails, you can switch to the standby cluster to reduce the impact on your workloads. After the active cluster recovers, you can use LTS to synchronize the incremental data from the standby cluster to the active cluster.
Historical data storage in ApsaraDB RDS databases
In scenarios where historical data, such as transaction orders, is stored, performance bottlenecks may occur in ApsaraDB RDS databases due to the ever-increasing data size. Periodic data archiving or sharding is complicated and causes high costs. LTS lets you synchronize data from ApsaraDB RDS to LindormTable in real time to separate hot data from cold data. LindormTable supports automatic horizontal scaling, high-concurrency queries, multi-dimensional indexing, and lightweight analysis. Lindorm Streams lets you track data changes in sequence. LTS also lets you synchronize data from LindormTable to other analytics systems for complex data analysis.