Simple Log Service (SLS) collects monitoring data from all clusters as logs, normalizes it into a unified format, and routes it to downstream systems for real-time or offline analysis. This topic explains how SLS fits into a monitoring architecture and how to set up the pipeline end to end.
When your infrastructure spans multiple clusters and heterogeneous systems, a centralized log pipeline solves two practical problems: you no longer manage collection agents per cluster, and every downstream consumer—whether a real-time stream processor like Storm or a query engine like Elasticsearch—reads data in the same format.
SLS supports horizontal scaling by adding shards when traffic spikes, and lets you onboard tens of thousands of machines through a single configuration rather than per-machine setup.
What SLS gives your monitoring pipeline
|
What you can do |
How SLS supports it |
|
Manage machines at scale |
Install Logtail once per server; all subsequent configuration is done in SLS. |
|
Push config changes everywhere |
Define which log files to collect in SLS; the configuration propagates to all servers automatically. |
|
Normalize heterogeneous log data |
All collected data is converted to the SLS data model, so downstream consumers get a consistent structure. |
|
Handle traffic spikes |
Add shards to scale read/write throughput without downtime. |
Once logs are in SLS, downstream systems consume them via API. Spark and Storm can process the data in real time; Elasticsearch, Tablestore, and MaxCompute can store or query it. Because ingestion happens once and multiple systems can consume independently, monitoring is the most common use case.
The diagram below shows the full architecture: Logtail collects logs from servers into SLS, middleware pulls the data using the PullLogs API, a Storm cluster runs the computing rules, and results are written to Tablestore for display or alerting.
Figure 1. Architecture of the monitoring system
Build the monitoring system
-
Collect monitoring data.
Configure log collection in SLS so that logs from your servers and middleware are collected and forwarded to SLS.
-
Consume log data via API.
Use the SDK to call the PullLogs operation, which pulls log data from SLS in batches and streams it to your real-time computing system.
-
Set up a real-time computing system.
Configure Storm or another real-time computing engine with your computing rules. Select the metrics to compute, then write the results to Tablestore.
-
Display and alert on monitoring data.
Read the results from Tablestore to display them on a front-end dashboard. Configure alerts based on threshold conditions in the data.