As enterprises expand their global businesses, a key challenge emerges. It involves efficiently, economically, and reliably collecting logs from overseas applications and infrastructure into Alibaba Cloud Simple Log Service (SLS) for analysis and monitoring.
This article focuses on the application of high-performance log collection agents (iLogtail and LoongCollector) from Alibaba Cloud in overseas scenarios. It delves into how to design optimal network access links for different deployment environments, including on-premises data centers, cross-cloud platforms, and Alibaba Cloud environments. We recommend LoongCollector first because it is more reliable, especially in multi-target transmission scenarios. In this article, a variety of network solutions are analyzed in detail, including direct internet connection, Global Accelerator (GA) optimization, Alibaba Cloud internal network, leased line, CEN, and VPN access.
In addition, this article highlights cost optimization strategies, including using CloudLens for SLS to diagnose usage and migrating public links to private networks to reduce cost. At the same time, the article also elaborates on two core multi-target transmission configurations: one is to implement cross-region disaster recovery (data redundancy) by using agent dual-write, and the other is to implement multi-region or multi-target log distribution (on-demand routing).
This article aims to provide comprehensive practical guidance and configuration reference for enterprises to build a stable, low-cost, and highly available global log system.
With the expansion of global business, log data is the cornerstone of observability, troubleshooting, and compliance auditing. The unified collection and analysis of log data has thus become crucial. However, enterprises generally face the following key challenges when connecting logs distributed around the world (including local data centers, third-party cloud service providers, and Alibaba Cloud overseas regions) to Alibaba Cloud SLS. These challenges are directly related to access link selection, overall cost control, and system high availability:
Link quality and stability challenges
Pressure of cost control
High availability and disaster recovery requirements
Multiple environments and compliance complexity
In dealing with the above challenges, the selection of an appropriate collection agent is crucial. As the agents recommended by Alibaba Cloud, both iLogtail and LoongCollector have powerful basic capabilities. However, LoongCollector provides further enhancements in terms of reliability and some advanced features.
● Lightweight and efficient: With a C++ core, they have low resource usage (CPU, memory) and minimal impact on business server performance.
● All-purpose collection: They support multiple data sources such as file logs, container logs (stdout, stderr, and files), Syslog, and HTTP.
● Powerful processing (processor plug-ins): They support preprocessing such as parsing (JSON, Regex, and delimiters), filtering, and desensitization of data on the agent, thereby reducing invalid data transmission from the source, optimizing costs, and improving backend processing efficiency.
● Transmission compression: The built-in LZ4 compression algorithm can significantly reduce network traffic.
● Highly reliable transmission: They are equipped with mechanisms such as local disk caching, resumable transmission, retry upon failure, and traffic shaping. These mechanisms effectively handle network fluctuations and ensure that data is "neither lost nor duplicated."
● Flexible output and multi-target: They support sending data to SLS, and a single agent instance can be configured to send the same piece of data to multiple target SLS endpoints simultaneously (for dual-write disaster recovery or migration).
● Cloud-native and ecosystem integration: They are deeply integrated with Alibaba Cloud ECS, container services such as ACK and ASK, and support convenient deployment (DaemonSet, Sidecar) and configuration management (CRD) in Kubernetes environments.
Enhanced reliability - network anomaly isolation:
Conclusion: Although iLogtail remains a powerful agent, for users with high reliability requirements, especially those involving complex scenarios such as multi-region log transmission, it is strongly recommended to prioritize LoongCollector, as it offers superior stability and network fault tolerance. LoongCollector is in the process of canary release. If you have more questions, please contact us through tickets.
Recommended versions:
Depending on the business deployment location, network conditions, cost budget, and requirements for latency, stability, and security, you can choose different access link solutions:
Solution 1: Directly connect to public endpoints
Solution 2: Optimize the Internet by GA
Solution 3: Private network access for services in the same region on Alibaba Cloud (optimal cost and performance)
Solution 4: In hybrid cloud or multi-cloud scenarios, connect to an intranet through a leased line or a Cloud Enterprise Network (CEN)
Solution comparison summary:
| Solution | Network Cost | Latency | Stability | Data Security | Deployment Complexity | Recommended Scenarios |
|---|---|---|---|---|---|---|
| Direct Internet Connection | Medium (Internet traffic) | High | Low | Medium | Low | Scenarios with low budgets and low requirements. |
| Global Accelerator (GA) | High (GA traffic fee) | Medium | Medium | Medium | Low | Cross-border transmission (especially to the mainland) with high requirements for real-time performance and stability. |
| Private Network in the Same Region | Low | Low | High | High | Low | Strongly recommended: Services and SLS are in the same region on Alibaba Cloud. |
| CEN/Leased Line | Extremely high | Low | High | High | High | Local data center or multi-cloud private network interconnection; core services; high security, stability, and real-time requirements; sufficient budget. |
In addition to selecting appropriate network links, the following strategies can further optimize costs:
As your business continues to grow, the amount of log data that is added to SLS increases. However, sometimes unexpected log volume or Internet traffic surges may occur. This is often due to the application printing too much redundant information, or the imprecise collection configuration (for example, unnecessary debug logs are collected, logs are collected repeatedly, or the collection path range is overly large to collect irrelevant log files). These situations can not only cause unimportant logs to occupy a large amount of resources, but also significantly increase the associated network transmission and storage costs.
To effectively monitor and diagnose these issues, Alibaba Cloud Simple Log Service (SLS) provides CloudLens for SLS. This service provides you with a centralized view that clearly displays the core resource consumption metrics of all projects in your account, including:
Log write volume:
Outbound Internet traffic:
GA traffic:
By analyzing the reports provided by CloudLens (see View CloudLens Data Reports), you can quickly locate the projects or specific periods whose traffic usage is abnormal (excessive or sudden increase). This helps you detect and troubleshoot unexpected log writing behaviors in a timely manner. For example:
Based on these insights, you can take targeted optimization measures to control costs and improve resource usage efficiency. If you need further assistance while analyzing CloudLens data or troubleshooting specific issues, feel free to contact Alibaba Cloud technical support by submitting a ticket at any time.
References: How to Use CloudLens for SLS to Help You Analyze Resource Usage and View CloudLens for SLS Data Reports
Log data, especially raw text logs, often contains a large amount of redundant information. Direct transmission over the network consumes considerable bandwidth resources. Especially in scenarios where logs are transmitted across regions, across countries, or over the Internet, the high cost of network bandwidth is an important part of the total cost of ownership (TCO) of the log system. To mitigate this issue, both LoongCollector and iLogtail, the data collection agents of Alibaba Cloud Simple Log Service (SLS), support client-side compression before data transmission.
In many scenarios, not all generated logs have the same analytical value. For example, DEBUG-level logs may not be necessary in production, or some frequently printed health check logs may not be meaningful for core business analysis. Uploading these low-value or redundant logs to SLS needlessly increases network transmission cost, SLS indexing and storage costs, and may interfere with the rapid location of critical information.
iLogtail and LoongCollector provide powerful processor plug-ins, which allow you to preprocess and filter collected logs on the agent. This way, unnecessary log entries are discarded before transmission.
You can use the following methods to filter logs on the agent based on your business requirements:
Processor filter plug-in:
SPL-based filtering:
References:
For scenarios where public Internet access was used in the past and now there is a desire to switch to a private network with lower costs and better performance, a smooth transition is often required to avoid monitoring interruptions. This typically involves configuring the agent for dual-write during the migration period. This section describes the following two typical scenarios.
For specific steps, please refer to [Best Practices] Are You Still Transmitting Data Across Borders? Compliance Governance for Cross-border Data Transmission - Local Migration of Logtail Data Without Service Interruption
In addition to sending the same copy of data to multiple regions for disaster recovery or migration (as in the dual-write scenario described in Section 4.3 and Part 5), there is another common requirement:
Different logs are sent to different destinations: Logs of different types or different sources from the same server or Kubernetes nodes are sent to SLS in different regions, respectively, for processing and analysis.
For example:
Business machine group in Singapore
iLogtail and LoongCollector fully support this scenario, with the core implementation method being to configure multiple endpoints for the same agent instance. For specific steps, please refer to the following.
1. Suppose an ECS instance in Singapore needs to write logs to both the Shanghai and Singapore regions simultaneously.
2. On the SLS console, create Project A (located in the Shanghai region) and Project B (located in the Singapore region).
3. Create machine groups under Project A and Project B respectively.
4. On the server where the client is located, configure the ilogtail_config.json file of iLogtail or LoongCollector to support dual write.
The following provides a configuration sample of ilogtail_config.json for iLogtail or LoongCollector. Among them, LoongCollector is also compatible with the ilogtail_config.json configuration sample of iLogtail.
ilogtail_config.json configuration sample of iLogtail
{
...
"config_server_address":"http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
"config_server_address_list": [
"http://cn-shanghai.log.aliyuncs.com"
],
"data_server_list": [
{
"cluster": "ap-southeast-1",
"endpoint":"ap-southeast-1-intranet.log.aliyuncs.com"
},
{
"cluster": "cn-shanghai",
"endpoint":"cn-shanghai.log.aliyuncs.com" // If GA is required, replace this with log-global.aliyuncs.com
}
],
...
}ilogtail_config.json configuration sample of LoongCollector
{
...
"primary_region" : "ap-southeast-1",
"config_servers" :
[
"http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
"http://logtail.cn-shanghai.log.aliyuncs.com"
],
"data_servers" :
[
{
"region" : "ap-southeast-1",
"endpoint_list": [
"ap-southeast-1-intranet.log.aliyuncs.com"
]
},
{
"region" : "cn-shanghai",
"endpoint_list": [
"cn-shanghai.log.aliyuncs.com" // If GA is required, replace this with log-global.aliyuncs.com
]
}
],
...
}5. Restart iLogtail or LoongCollector.
1. Suppose an ACK cluster in Singapore needs to write logs to both the Shanghai and Singapore regions simultaneously.
2. On the SLS console, create Project A (located in the Shanghai region) and Project B (located in the Singapore region).
3. Create machine groups under Project A and Project B respectively.
4. In the ACK cluster, configure the ilogtail_config.json file of iLogtail or LoongCollector to support dual write.
For a specific configuration example of the ilogtail_config.json for iLogtail or LoongCollector, see the example provided in the foregoing ECS.
a) Create a ConfigMap.
b) Modify the deployment of logtail-ds or loongcollector-ds.
5. Restart logtail-ds or loongcollector-ds.
1. Observe the heartbeat status of the machine groups in Project A and Project B, and confirm that the heartbeat of both machine groups is functioning properly and that the machines in the two groups are consistent.
2. Check if data is reported under Logstore A of Project A and Logstore B of Project B.
Note: CloudLens for SLS is recommended for collection status monitoring.
Dual-write collection is a high-availability strategy. It means that through configuration, the iLogtail or LoongCollector agent on the client can send the same log data in real time and in parallel to two independent Alibaba Cloud SLS projects located in different regions (or different zones in the same region). If a zone of SLS fails, you can use this method to switch to another available SLS region.
First, refer to the fifth chapter to configure a multi-region collection environment, and then enable the collection configuration in your disaster recovery project to allow multiple file collections, so that logs can be written to the disaster recovery project.
How We Built an Observability Agent 10x Faster Than Open Source Alternatives
626 posts | 54 followers
FollowAlibaba Cloud Native Community - August 11, 2025
Alibaba Cloud Community - October 9, 2022
Alibaba Clouder - July 15, 2020
Alibaba Cloud Native Community - July 26, 2022
Alibaba Cloud Native Community - July 31, 2025
Alibaba Cloud Community - August 12, 2024
626 posts | 54 followers
Follow
Best Practices
Follow our step-by-step best practices guides to build your own business case.
Learn More
Application High Availability Service
Application High Available Service is a SaaS-based service that helps you improve the availability of your applications.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn MoreMore Posts by Alibaba Cloud Native Community