You can consume logs in the following three methods after they are collected to Log Service LogHub.
|Real-time consumption (LogHub)||Stream computing and real-time computing||Real-time (< 10 ms)||365 days (contact us for a longer storage period)|
|Query and analysis (LogSearch/Analytics)||Online query and analysis||Real-time (less than one second in 99.99% cases)||365 days (contact us for a longer storage period)|
|Shipping and storage (LogShipper)||Full log storage for offline analysis||5–30 minutes||Depends on the storage system|
Logs are consumed after being written. Both log consumption and log query require the capability of reading logs. For the difference between log consumption and log query, see Differences between log consumption and log query. Logs in a shard are consumed as follows.
- Obtain a cursor based on conditions such as time, Begin, and End.
- Read logs by using the cursor and step, and return the next cursor.
- Continuously move the cursor to consume logs.
Besides the basic APIs, Log Service provides many methods to consume logs, such as SDKs, Storm spout, Spark Streaming client, Flink connector, consumer library, and Web console.
- Use Spark Streaming client to consume logs.
- Use Storm spout to consume logs.
- Use Flink connector, including Flink consumer and Flink producer to consume logs.
- Use consumer library to consume logs. The consumer library is an advanced mode for LogHub consumers, which provides a lightweight computing framework and solves the issue of automatic shard allocation and order preservation when multiple consumers consume a Logstore at the same time. For more information, see Consumer library.
- Use SDKs to consume logs. Log Service provides SDKs in multiple languages (Java and Python) that support the log consumption APIs. For more information about SDKs, see Log Service SDKs.
- Use cloud products to consume logs:
- Query logs in the Log Service console: See Query logs.
- Query logs by using Log Service SDKs/APIs: Log Service provides RESTful APIs that are implemented based on HTTP protocol. The Log Service APIs also provide a full-featured log query API. For more information, see Log Service APIs.
- Ship logs to OSS: Store logs for a long term or use E-MapReduce to analyze logs.
- Ship logs to Table Store: Use NoSQL to store logs.
Secure Log Service: Log Service interconnects with cloud security products and uses ISV to consume logs of cloud products.