All Products
Search
Document Center

Lindorm:Introduction to LTS (formerly known as BDS)

Last Updated:Aug 24, 2022

Lindorm Tunnel Service (LTS) is a data ecosystem service that is customized based on the characteristics of business scenarios in which Lindorm is used. LTS provides easy-to-use capabilities, including data exchange, processing, and change tracking. These capabilities enable you to migrate data, track real-time data changes, dump data to data lakes, and synchronize data from data warehouses to Lindorm databases. These capabilities also enable you to apply multi-active redundancy based on units and implement backup and restoration. This way, LTS provides an all-in-one data ecosystem service for Lindorm.

Core features

  • Cloud native distributed system: LTS is a distributed system that is deployed based on Elastic Compute Service (ECS). LTS features excellent horizontal scalability and allows you to configure resources based on your business requirements.

  • Ease of use: LTS allows you to configure data migration, import, change tracking, and archiving tasks. For example, to create a data migration task, you need to only specify the migration source, the migration destination, and the columns to be synchronized. LTS automatically replicates schemas, full data, and incremental data based on the settings.

  • High security and reliability: LTS minimizes the impact on the source and destination systems and minimizes the impact of potential failures due to incompatibility. Before a task is started, LTS prechecks the network connectivity and security. While the task is running, LTS monitors the synchronization latency and the storage usage of the destination cluster in real time. LTS also implements throttling and reports alerts based on the monitoring data. After the task is complete, LTS verifies data.

  • Cost-efficiency: LTS is an optimized service based on open source systems, such as Apache HBase, Apache Phoenix, and Apache Cassandra. LTS allows you to process data at the physical file level. This is 10 times more efficient than traditional data replication. LTS also provides optimized CPU, cache, memory, and network I/O capabilities. This enables LTS to provide cost-effective tunnels and helps reduce your costs of data transfer and processing.

Features

Feature

Scenario

References

Data migration between HBase and LindormTable

Seamless data migration between previous clusters and new clusters, cluster upgrades, online and offline workload decoupling, primary/secondary disaster recovery, and active geo-redundancy.

Synchronize full and incremental data

RDS -> Lindorm

Online and offline workload decoupling and historical data archiving.

Synchronize full data and incremental data from ApsaraDB RDS

MaxCompute/Hive -> Lindorm

Accelerates offline data queries, and returns analysis results such as details and metrics from data warehouses to Lindorm in batches for online data queries.

To be supplemented. For more information, contact online support.

Data export from Lindorm to MaxCompute (previously known as Open Data Processing Service (ODPS))

Allows you to export historical data and incremental data.

Export full data to MaxCompute and Archive incremental data to MaxCompute

Subscription to real-time data in LogHub

Allows you to subscribe to real-time data from LogHub and consume the data in Lindorm.

Import incremental data from Log Service

Change tracking

Allows you to subscribe to real-time incremental data in Lindorm.

Lindorm Stream

Log lifecycle management

  • If log data is not consumed after you enable the log subscription feature, the log data is retained for 48 hours by default. After the period expires, the subscription is automatically canceled and the retained data is automatically deleted.

  • Log data may fail to be consumed if your LTS cluster is released while your task is still running or if your synchronization task is suspended.

  • You can enable the log subscription feature for the following types of tasks in Lindorm: incremental synchronization, data archiving, data backup, and data subscription.

Common scenarios

  • Cluster migration

    • Usage scope

      • Data migration from HBase to Lindorm.

      • Cluster network switch. For example, the network type is changed from the classic network to a virtual private cloud (VPC).

      • Data center migration across regions.

      • Workload decoupling.

    • Features

      • When data is being migrated, no service downtime is caused. In one task, LTS can migrate historical data and synchronize real-time incremental data.

      • When data is being migrated, LTS does not interact with the source HBase or Lindorm cluster. LTS reads data only from the HDFS of the source cluster. This minimizes the impact on the online business that runs on the source cluster.

      • In most cases, compared with data migration at the API layer, data replication at the file layer can help you reduce more than 50% of the data usage.

      • High efficiency is provided. Each node can migrate data at a rate of up to 100 MB/s. You can add nodes for horizontal scaling to migrate terabytes or even petabytes of data.

      • High stability is provided. LTS supports a robust mechanism of rerunning failed tasks. LTS monitors the synchronization rates and progress of tasks in real time, and reports alerts when tasks fail.

      • Data accuracy is ensured. LTS verifies synchronized data.

      • Automatic schema synchronization is supported. This ensures consistent partitions.

  • Online and offline workload decoupling

    LTS allows you to synchronize online business data in real time to the HDFS or OSS storage system. LTS can work with components of big data services, such as Spark and MapReduce, to analyze data. This ensures that online business queries are not affected.

  • Primary/secondary disaster recovery

    LTS supports two-way data synchronization between a primary cluster and a secondary cluster. When the primary cluster fails, you can switch your workloads to the secondary cluster to reduce the impact on the workloads. After the primary cluster recovers, you can use LTS to synchronize the incremental data from the secondary cluster to the primary cluster.

  • Historical data storage in ApsaraDB RDS databases

    In scenarios where historical data, such as transaction orders, is stored, performance bottlenecks may occur in ApsaraDB RDS databases due to the ever-increasing data size. Periodic data archiving or sharding is complicated and causes high costs. LTS allows you to synchronize data from ApsaraDB RDS to LindormTable in real time to separate hot data from cold data. LindormTable supports automatic horizontal scaling, high-concurrency queries, multi-dimensional indexing, and lightweight analysis. Lindorm Streams allows you to track data changes in sequence. LTS also allows you to synchronize data from LindormTable to other analytics systems for complex data analysis.