The following figure shows the architecture of Log Service.

Figure 1. Architecture
architecture

Logtail

The Logtail agent collects logs. It has the following characteristics:
  • Non-intrusive file-based log collection
    • Logtail only reads log files.
    • Log collections are not intrusive.
  • High security and reliability
    • Logtail can rotate log files without data loss.
    • Logtail supports local caching.
    • Logtail retries when network exceptions occur.
  • Ease of use and management
    • Logtail supports configurations by using the API or SDK.
    • Logtail supports configurations in the console.
  • Complete self-protection mechanism
    • Logtail monitors CPU and memory resources that it consumes in real time.
    • Logtail allows you to set an upper limit on the resource it consumes.

Frontend servers

Frontend servers are built on LVS and NGINX. They have the following characteristics:
  • Support for HTTP and REST
  • Scale-out
    • The processing capabilities can be increased quickly when traffic increases.
    • Frontend servers can be added.
  • High throughput and low latency
    • Asynchronous processing: If an exception occurs when a single request is sent, other requests are not affected.
    • LZ4 compression: The processing capabilities of individual servers are increased while network bandwidth consumption is reduced.

Backend servers

The backend service is a distributed process deployed on multiple servers. The service performs storage, indexing, and queries on Logstore data in real time as well as ships the data to MaxCompute. The overall characteristics of the backend service are as follows:
  • High data security
    • Each log entry is stored in three copies on different servers.
    • Data is automatically resumed in the event of disk damage or server downtime.
  • Stable service
    • Logstores are automatically migrated in the event of process crashes or server downtime.
    • Automatic load balancing ensures that traffic is distributed evenly among servers.
    • Strict resource quota limits prevent incorrect or unexpected operations of a single user from affecting other users.
  • Scale-out
    • A shard is the basic unit for scale-out.
    • You can add shards to increase throughput based on your needs.