Architecture

Last Updated: Nov 29, 2017

The Log Service system architecture is shown in the following diagram.

1

Logtail

Logtail helps you quickly collect logs through the following features:

  • Non-invasive log collection based on log files
    • Only read files.
    • Unobtrusive during reading process.
  • Secure and reliable
    • Supports file rotation, so data is not lost.
    • Supports local caching.
    • Provides network exception retry mechanism.
  • Convenient management
    • Web client.
    • Visualization configuration.
  • Comprehensive self-protection
    • Real-time monitoring of process CPU and memory.
    • Consumption and restrictions on CPU/memory usage.

Frontend servers

Frontend machines are built using LVS+Nginx. Its features are as follows:

  • HTTP and REST protocols
  • Horizontal scaling
    • Support horizontal scaling When traffic increases
    • Frontend machines can be quickly added to improve processing capabilities.
  • High throughput, low latency
    • Pure asynchronous processing, a single request exception will not affect other requests.
    • Lz4 compression is adopted to increase the processing capabilities of individual machines and reduce network bandwidth consumption.

Backend servers

The backend is a distributed process deployed on multiple machines. It provides real-time Logstore data persistence, indexing, query, and shipping to OSS. The features of the overall backend service are as follows:

  • High data security
    • Each log you write is saved in triplicate.
    • Data is automatically recovered in case of any disk damage or machine downtime.
  • Stable service
    • Logstores automatically migrate in case of a process crash or machine downtime.
    • Automatic server load balancing ensures that traffic is distributed evenly among different machines.
    • Strict quota restrictions that prevent abnormal behavior of a single user from affecting other users.
  • Horizontal scaling
    • Horizontal scaling is performed using shards as the basic unit.
    • You can dynamically add shards as needed to increase throughput.
Thank you! We've received your feedback.