High-availability solutions for client data collection

Simple Log Service provides two high-availability solutions for client data collection: dual-write and data transformation-based replication with write switchover.

Background

In a single-cluster environment, Simple Log Service stores three replicas of each piece of data on different physical disks across different servers, using a distributed storage algorithm. This triplicate mechanism ensures data reliability, strong consistency among replicas, and high availability against single hardware failures.

However, a single cluster may still experience service interruptions in extreme scenarios. To mitigate this risk, Simple Log Service provides two active geo-redundancy solutions for client data collection.

Solution comparison

Item	Solution 1: Dual-write	Solution 2: Data transformation-based replication and write switchover
Deployment complexity	Low	Requires additional deployment for data transformation and write switchover.
Client resource usage	Twice that of Solution 2.	Low
Data consistency	Strong	A certain percentage of data may be duplicated or lost during a switchover.

Network recommendations

Logtail requires reliable network connectivity for client data collection. We recommend that you use a leased line for cross-region data transmission.

Solution 1: Dual-write

The dual-write solution writes client data to two Simple Log Service Logstores in different regions simultaneously.

Under normal conditions, data is available in both Logstores in real time. If either Logstore fails, data in the other Logstore remains available.

Procedure:

1. In the Simple Log Service console, create a Project and a Logstore.

   1. Create Project A and Logstore A in Region A.

   2. Create Project B and Logstore B in Region B.

2. On the client server, configure the logtail_config.json file for Logtail to support writing to two regions.

   For example, if Region A is China (Shanghai) and Region B is China (Beijing), add the config_server_address_list and data_server_list configuration items. Set the domain name for China (Beijing) as the second address and keep other settings unchanged.

   {
       ...
       "config_server_address":"logtail.cn-shanghai-intranet.log.aliyuncs.com",
       "config_server_address_list": [
           "cn-shanghai-intranet.log.aliyuncs.com",
           "cn-beijing-intranet.log.aliyuncs.com"
       ],
       "data_server_list": [
           {
               "cluster": "cn-shanghai",
               "endpoint":"cn-shanghai-intranet.log.aliyuncs.com"
           },
           {
               "cluster": "cn-beijing",
               "endpoint":"cn-beijing-intranet.log.aliyuncs.com"
           }
       ],
       "cpu_usage_limit": 2.0,
       ...
   }

3. In the Simple Log Service console, create Logtail configurations.

   Important

   • You can set "accept_multi_config": true in the Logtail startup parameters, or add {"force_multiconfig": true} in the Advanced Configurations section when you create a Logtail configuration. This lets you collect logs from a single file multiple times. For more information, see How to collect multiple copies of logs from a file.

   • When you modify a Logtail configuration, you must modify the configurations in both Projects to ensure consistency.

   1. In Project A, create a Logtail configuration in Logstore A.

   2. In Project B, create a Logtail configuration in Logstore B.

Solution 2: Data transformation-based replication and write switchover

This solution collects client data into Logstore A in one region and uses data transformation to replicate the data to Logstore B in another region in real time. If Logstore A fails, you can switch the write destination to Logstore B.

Under normal conditions, data is available in both Logstores in real time. If Logstore A fails, you can switch data collection to Logstore B to maintain real-time availability. If Logstore B fails, data in Logstore A remains available.

Procedure:

1. In the Simple Log Service console, create a Project and a Logstore.

   1. Create Project A and Logstore A in Region A.

   2. Create Project B and Logstore B in Region B.

2. On the client server, configure the logtail_config.json file for Logtail to support writing to two regions.

   For example, if Region A is China (Shanghai) and Region B is China (Beijing), add the config_server_address_list and data_server_list configuration items. Set the domain name for China (Beijing) as the second address and keep other settings unchanged.

   {
       ...
       "config_server_address":"logtail.cn-shanghai-intranet.log.aliyuncs.com",
       "config_server_address_list": [
           "cn-shanghai-intranet.log.aliyuncs.com",
           "cn-beijing-intranet.log.aliyuncs.com"
       ],
       "data_server_list": [
           {
               "cluster": "cn-shanghai",
               "endpoint":"cn-shanghai-intranet.log.aliyuncs.com"
           },
           {
               "cluster": "cn-beijing",
               "endpoint":"cn-beijing-intranet.log.aliyuncs.com"
           }
       ],
       "cpu_usage_limit": 2.0,
       ...
   }

3. In the Simple Log Service console, create a Logtail configuration in Logstore A of Project A.

   Important

   You can set "accept_multi_config": true in the Logtail startup parameters, or add {"force_multiconfig": true} to the Advanced Configurations section when you create a Logtail configuration. This allows logs from a single file to be collected multiple times. For more information, see How to collect multiple copies of logs from a file.

4. Use Terraform or an API to back up the Logtail configuration from Project A. Modify the backup to create a Logtail configuration for Project B that can be used for failover.

5. Create a data transformation task to synchronize Logstore data in real time. For more information, see Replicate Logstore data and Transmit data across regions.

6. If an error occurs in Logstore A, import the backup Logtail configuration from Step 4 into Logstore B.