edit-icon download-icon

Overview

Last Updated: Mar 20, 2018

You can consume logs in the following three methods after they are collected to Log Service LogHub.

Method Scenario Timeliness Storage period
Real-time consumption (LogHub) Stream computing and real-time computing Real-time (< 10 ms) 365 days (contact us for a longer storage period)
Query and analysis (LogSearch/Analytics) Online query and analysis Real-time (less than one second in 99.99% cases) 365 days (contact us for a longer storage period)
Shipping and storage (LogShipper) Full log storage for offline analysis 5–30 minutes Depends on the storage system

Real-time consumption

Consumption process

Logs are consumed after being written. Both log consumption and log query require the capability of reading logs. For the difference between log consumption and log query, see Differences between log consumption and log query. Logs in a shard are consumed as follows.

  1. Obtain a cursor based on conditions such as time, Begin, and End.
  2. Read logs by using the cursor and step, and return the next cursor.
  3. Continuously move the cursor to consume logs.

Consumption method

Besides the basic APIs, Log Service provides many methods to consume logs, such as SDKs, Storm spout, Spark Streaming client, Flink connector, consumer library, and Web console.

  • Use Spark Streaming client to consume logs.
  • Use Storm spout to consume logs.
  • Use Flink connector, including Flink consumer and Flink producer to consume logs.
  • Use consumer library to consume logs. The consumer library is an advanced mode for LogHub consumers, which provides a lightweight computing framework and solves the issue of automatic shard allocation and order preservation when multiple consumers consume a Logstore at the same time. For more information, see Consumer library.
  • Use SDKs to consume logs. Log Service provides SDKs in multiple languages (Java and Python) that support the log consumption APIs. For more information about SDKs, see Log Service SDKs.
  • Use cloud products to consume logs:

Query and analysis

See Overview of real-time query and analysis:

  • Query logs in the Log Service console: See Query logs.
  • Query logs by using Log Service SDKs/APIs: Log Service provides RESTful APIs that are implemented based on HTTP protocol. The Log Service APIs also provide a full-featured log query API. For more information, see Log Service APIs.

Shipping and storage

Others

Secure Log Service: Log Service interconnects with cloud security products and uses ISV to consume logs of cloud products.

Thank you! We've received your feedback.