Logs collected to the LogHub of the Log Service can be consumed in the following three methods:

Approach Scenario Real time Storage period
Real-time consumption (loghub) Stream computing and real-time computing Real-time Customize
Query and analysis (LogSearch/Analytics) Online query and analysis Real-time (less than one second in 99.99% cases) Customize
Shipping and storage (LogShipper) Full log storage for offline analysis 5–30 minutes Depends on the storage system

Real-time consumption

Logs are consumed after being written. Both log consumption and log query require the capability of reading logs. Logs in a shard are consumed as follows.
  1. Obtain a cursor based on a set of criteria such as time, Begin, and End.
  2. The system reads logs based on the cursor and step and returns the next cursor.
  3. Moves the cursor continuously to consume logs.
Besides the basic APIs, Log Service provides many methods to consume logs, such as SDKs, Storm spout, Spark Streaming client, Flink connector, consumer library, and Web console.
  • Use Spark Streaming Client to consume logs.
  • Use Storm Spout to consume logs.
  • Use Flink Connector, including Flink consumer and Flink producer to consume logs.
  • Use LogHub Consumer  Library  to consume logs.  The consumer library is an advanced mode for LogHub consumers, which provides a lightweight computing framework and solves the issue of automatic shard allocation and order preservation when multiple consumers consume a Logstore at the same time.
  • Use SDKs to consume logs. Log Service provides SDKs in multiple languages (Java and Python) that support the log consumption APIs. For more information about SDKs, see Log Service SDK.
  • Use cloud products to consume logs:

Query and analysis

Overview of real-time query and analysis:
  • Query logs in the Log Service console: See Query logs.
  • Query logs by using Log Service SDKs/APIs: Log Service provides RESTful APIs that are implemented based on HTTP protocol.  The Log Service APIs also provide a full-featured log query API.  For more information, see Log Service APIs.

Shipping and storage

Others

Secure Log Service: Log Service interconnects with cloud security products and uses ISV to consume logs of cloud products.