Logs collected to the LogHub of the Log Service can be consumed in the following three methods:
| Approach | Scenario | Real time | Storage period |
|---|---|---|---|
| Real-time consumption (loghub) | Stream computing and real-time computing | Real-time | Customize |
| Query and analysis (LogSearch/Analytics) | Online query and analysis | Real-time (less than one second in 99.99% cases) | Customize |
| Shipping and storage (LogShipper) | Full log storage for offline analysis | 5–30 minutes | Depends on the storage system |
Real-time consumption
Logs are consumed after being written. Both log consumption and log query require
the capability of reading logs. Logs in a shard
are consumed as follows.
Besides the basic APIs, Log Service provides many methods to consume logs, such as
SDKs, Storm spout, Spark Streaming client, Flink connector, consumer library, and
Web console.
- Obtain a cursor based on a set of criteria such as time, Begin, and End.
- The system reads logs based on the cursor and step and returns the next cursor.
- Moves the cursor continuously to consume logs.
- Use Spark Streaming Client to consume logs.
- Use Storm Spout to consume logs.
- Use Flink Connector, including Flink consumer and Flink producer to consume logs.
- Use LogHub Consumer Library to consume logs. The consumer library is an advanced mode for LogHub consumers, which provides a lightweight computing framework and solves the issue of automatic shard allocation and order preservation when multiple consumers consume a Logstore at the same time.
- Use SDKs to consume logs. Log Service provides SDKs in multiple languages (Java and Python) that support the log consumption APIs. For more information about SDKs, see Log Service SDK.
- Use cloud products to consume logs:
- Use CloudMonitor to consume logs: Monitoring scenario.
- Use E-MapReduce to consume logs: See Storm, Spark Streaming.
- Use Command-line tools CLI to consume logs.
Query and analysis
Overview of real-time query and analysis:
- Query logs in the Log Service console: See Query logs.
- Query logs by using Log Service SDKs/APIs: Log Service provides RESTful APIs that are implemented based on HTTP protocol. The Log Service APIs also provide a full-featured log query API. For more information, see Log Service APIs.
Shipping and storage
- Ship logs to OSS: Store logs for a long term or use E-MapReduce to analyze logs.Analysis log.
- Use function calculations for custom delivery.
Others
Secure Log Service: Log Service interconnects with cloud security products and uses ISV to consume logs of cloud products.