Simple Log Service:Analyze LoongCollector log collection latency

Symptoms

In the LoongCollector Pipeline Monitoring dashboard of CloudLens for SLS, view the Pipeline Overview report. The last column, Quota error count, shows the number of times LoongCollector failed to write to SLS within the queried time range due to exceeding server-side limits.

When LoongCollector is throttled, the Quota error count is highlighted in red.

Cause

SLS includes resource concepts such as Projects, LogStores, and Shards. To ensure reliable Quality of Service (QoS) for all tenants, each resource has associated resource throttling limits and solutions (i.e., quotas). Exceeding these limits can lead to failed data reads and writes or request exceptions.

In LoongCollector data collection scenarios, two common quota errors are:

• WriteQuotaExceed: This error indicates that the total volume of data or the number of write requests to a Project within a specific time period has exceeded the quota limit, causing subsequent requests to fail. This error is not necessarily caused by a single collection configuration but can result from the combined traffic from multiple collection configurations and LogStores.

• ShardWriteQuotaExceed: This error indicates that the total volume of data or the number of write requests to a single LogStore has exceeded its quota limit. The write capacity of a LogStore is determined by its Shards. For more information, see Manage Shards.

When LoongCollector fails to send data due to an exceeded quota, it retries continuously until the data is successfully sent. However, the total volume of data sent cannot exceed the quota limit. This creates a growing backlog of requests. As a result, each request waits longer to be sent, causing collection latency.

In the LoongCollector Pipeline Monitoring dashboard of CloudLens for SLS, you can check the Pipeline Send Status Details to determine whether the limit was exceeded for a Project or a LogStore Shard.

Solution

• If a Project write quota is exceeded, refer to "Manage resource quotas" in Manage Shards to adjust the Project write traffic limit and Project write QPS limit parameters. Then, submit a request to increase the quota.

  • As a general guideline, you can request to increase the current quota by about 50% each time.

• If a LogStore Shard write quota is exceeded, refer to the split shard and auto-split shard sections in Manage Shards to adjust the number of Shards for the LogStore.

  • Each Shard supports data writes of up to 5 MB/s or 500 requests per second, and data reads of up to 10 MB/s or 100 requests per second. These are soft limits. If they are exceeded, SLS continues to provide service but cannot guarantee service quality.

  • You can run the statement ((__topic__: loongcollector_metric and category: pipeline and source_ip:* and labels.project: * and labels.logstore: *))| select "labels.project" as project, "labels.logstore" as logstore, round(sum(flusher_in_size_bytes)/1024.0/1024.0, 2) as output_size_bytes group by project, logstore order by output_size_bytes desc limit all in the operational log LogStore of the corresponding Project to calculate the raw data write volume for different destination LogStores. To obtain the expected number of Shards, take the maximum value from the results, divide it by 5, and add a 20% buffer.

Alert configuration

Exceeding a quota is a critical issue. We recommend creating an alert rule.

You can check the self-monitoring data every 10 minutes for quota error messages and take action on the affected Projects or LogStores to prevent data reporting issues.

• Log type to monitor: operational log

• Filter condition: The value of the flusher_project_quota_error_total or flusher_shard_write_quota_error_total field is greater than 0. These fields correspond to the Project quota exceeded count and the LogStore Shard quota exceeded count, respectively.

• Display fields: Show the Project (labels.project), LogStore (labels.logstore), and collection configuration (labels.pipeline_name) that experienced quota errors within a specific time period.

• Query interval: 10 minutes. The alerts use self-monitoring data whose __topic__ is loongcollector_metric, which has a reporting interval of 10 minutes.

• Sample query

__topic__: loongcollector_metric and (flusher_project_quota_error_total>0 or flusher_shard_write_quota_error_total>0) | select "labels.project" as project, "labels.logstore" as logstore, split_part("labels.pipeline_name", '$', 2) as pipeline_name, sum(flusher_project_quota_error_total) as project_quota_error_total, sum(flusher_shard_write_quota_error_total) as shard_write_quota_error_total group by project, logstore, pipeline_name having project_quota_error_total>0 or shard_write_quota_error_total>0 order by pipeline_name limit all

Symptoms

In the LoongCollector Pipeline Monitoring dashboard of CloudLens for SLS, scroll down to Pipeline Send Status Details. If the Send Request Retry Rate is high while both the Project write quota exceeded count and the LogStore Shard write quota exceeded count are 0, it indicates that there are no quota-related errors. Instead, the high retry rate suggests that poor network quality is preventing timely data reporting and causing collection latency.

Cause

Internally, LoongCollector uses a pipeline for data reading, processing, and sending. After data is read from a file, it is sequentially passed to a processing queue and then to a sending queue. If the sending capacity of LoongCollector is limited due to poor network quality, large data volume, or other reasons, data cannot be sent to SLS promptly. This not only delays data querying in SLS but also causes data to accumulate in the sending queue. This accumulation creates back pressure on the processing queue and reading threads, further increasing collection latency.

Solution

1. Check your local network quality. If your business server and SLS Project are in different regions (for example, one in the Chinese mainland and one overseas), you cannot use the internal network. Transmitting data over the public internet may cause high network latency and instability. In this case, you can use Transfer Acceleration for better network stability and performance.

2. LoongCollector can speed up data sending by increasing the sending concurrency. To modify the startup parameters, see Set startup parameters. You can focus on adjusting the sending concurrency send_request_concurrency and the maximum sending traffic max_bytes_per_sec.

Alert configuration

Configure an alert for a high send request retry rate.

• Log type to monitor: operational log

• Filter condition: Filter for metrics related to the LoongCollector collection configuration.

• Display fields: Show the Project, LogStore, and collection configuration that have a high retry rate within a specific time period.

• Query interval: 10 minutes. Metrics are reported every 10 minutes. A high retry rate within a 10-minute interval indicates that LoongCollector may be experiencing collection latency.

• Sample query

((__topic__: loongcollector_metric and category: pipeline and source_ip:* and labels.pipeline_name:* and labels.project: *))| select "labels.project" as project, split_part("labels.pipeline_name", '$', 2) as pipeline_name,  (sum(flusher_send_done_total)-sum(flusher_success_total))*1.0/(sum(flusher_send_done_total)) as "send_retry_rate" group by project, pipeline_name having "send_retry_rate">0.1 and "send_retry_rate"<=1 order by "send_retry_rate" desc limit all 

Symptoms

First, confirm that SLS server-side write quotas are not exceeded and that there are no bottlenecks in LoongCollector's data sending.

• Check for collection bottlenecks

  Open Exception Monitoring in CloudLens for SLS, view All Error Messages in the Collection Exception Monitoring section, and filter for the PROCESS_QUEUE_BUSY_ALARM alert type. A large number of these alerts indicates a processing performance bottleneck in LoongCollector that causes collection latency. In the CloudLens for SLS console, select Report Center > Collection Monitoring, and click the Exception Monitoring tab. In the Collection Exception Monitoring section, view all error messages. If PROCESS_QUEUE_BUSY_ALARM appears in the Alert Type column, it indicates that the log processing queue is full (Alert details: logprocess queue is full, put modify event to event queue again).

• Identify collection configurations with high processing time

  In the LoongCollector Pipeline Monitoring dashboard of CloudLens for SLS, you can view the Pipeline Processing Time Percentage report to identify collection configurations with high processing times.

Cause

LoongCollector delivers excellent collection performance. For more information, see Log Collection Agent Performance Comparison—In-depth Performance Review of LoongCollector. A single thread using only native plugins can process data at over 100 MB/s. However, if a collection configuration uses extended plugins, complex regular expressions, or a chain of multiple plugins, LoongCollector may still reach a performance bottleneck when the data volume is large. In this case, you should first consider replacing extended plugins with SPL plugins or native plugins to improve processing performance. If performance is still insufficient, you need to adjust LoongCollector's internal processing thread count and CPU usage limit to process data more efficiently.

Solution

• First, replace all extended plugins with native plugins or use Logtail SPL to parse logs to significantly improve processing performance. Most commonly used extended plugins, such as those for regular expression parsing or JSON expansion, have corresponding native plugins. For more information, see the Processor Plugin List.

• If processing performance is still insufficient after you have replaced as many extended plugins as possible with native plugins, you can accelerate data processing in LoongCollector by increasing the CPU usage limit and the number of processing threads. See Set startup parameters to modify the startup parameters for LoongCollector, and focus on adjusting cpu_usage_limit and process_thread_count.

Alert configuration

Configure an alert for the PROCESS_QUEUE_BUSY_ALARM error. Note that an occasional PROCESS_QUEUE_BUSY_ALARM error may be caused by a sudden burst of data written to a log file and does not cause collection latency. Only persistent errors indicate that LoongCollector has insufficient processing capacity.

• Log type to monitor: operational log

• Filter condition: alarm_type is PROCESS_QUEUE_BUSY_ALARM, which indicates that the processing queue is full.

• Display fields: Show the Project, LogStore, and collection configuration that are experiencing slow processing errors within a specific time period.

• Query interval: 5 minutes. The alert interval is 30 seconds. If more than five errors occur within five minutes, it indicates that LoongCollector is experiencing collection latency due to insufficient processing capacity.

• Sample query

__topic__: logtail_alarm and alarm_type: PROCESS_QUEUE_BUSY_ALARM  | SELECT project, logstore, split_part(config_name, '$', 2) as "collection_configuration", count(1) as cnt from log group by project, logstore, config_name  having cnt >5 order by cnt desc limit all