Product architecture

Last Updated: Jul 25, 2017

The Log Service system architecture is shown in the following diagram.

1

Logtail

Logtail helps you quickly collect logs through the following features:

  • Non-invasive log collection based on log files
    • Read-only files.
  • Secure and reliable
    • Supports file rotation, so data are not lost.
    • Supports local caching.
    • Provides network exception retry mechanism.
  • Convenient management
    • Web client visualization configuration.
  • Comprehensive self-protection
    • Real-time monitoring of process CPU and memory consumption and restrictions on CPU/memory usage.

Frontend servers

Frontend machines are built using LVS+Nginx. Its features are as follows:

  • HTTP and REST protocols
  • Horizontal scaling
    • When traffic increases, frontend machines can be quickly added to improve processing capabilities.
  • High throughput, low latency
    • Pure asynchronous processing, a single request exception will not affect other requests.
    • Lz4 compression is adopted to increase the processing capabilities of individual machines and reduce network bandwidth consumption.

Backend servers

The backend is a distributed process deployed on multiple machines. It provides real-time Logstore data persistence, indexing, query, and shipping to MaxCompute (coming soon). The features of the overall backend service are as follows:

  • High data security
    • Each log you write is saved in triplicate.
    • Data are automatically recovered in case of any disk damage or machine downtime.
  • Stable service
    • Logstores automatically migrate in case of a process crash or machine downtime.
    • Automatic server load balancing ensures that traffic is distributed evenly among different machines.
    • Strict quota restrictions that prevent abnormal behavior of a single user from affecting other users.
  • Horizontal scaling
    • Horizontal scaling is performed using shards as the basic unit. You can dynamically add shards as needed to increase throughput.
Thank you! We've received your feedback.