Overview

You can consume logs in the following three methods after they are collected to Log Service LogHub.

Real-time consumption

Consumption process

Logs are consumed after being written. Both log consumption and log query require the capability of reading logs. For the difference between log consumption and log query, see Differences between log consumption and log query. Logs in a shard are consumed as follows.

1. Obtain a cursor based on conditions such as time, Begin, and End.
2. Read logs by using the cursor and step, and return the next cursor.
3. Continuously move the cursor to consume logs.

Consumption method

Besides the basic APIs, Log Service provides many methods to consume logs, such as SDKs, Storm spout, Spark Streaming client, Flink connector, consumer library, and Web console.

• Use Spark Streaming client to consume logs.
• Use Storm spout to consume logs.
• Use Flink connector, including Flink consumer and Flink producer to consume logs.
• Use consumer library to consume logs. The consumer library is an advanced mode for LogHub consumers, which provides a lightweight computing framework and solves the issue of automatic shard allocation and order preservation when multiple consumers consume a Logstore at the same time. For more information, see Consumer library.
• Use SDKs to consume logs. Log Service provides SDKs in multiple languages (Java and Python) that support the log consumption APIs. For more information about SDKs, see Log Service SDKs.
• Use cloud products to consume logs:
  • Use CloudMonitor to consume logs: Monitoring scenario.
  • Use E-MapReduce to consume logs: See Storm and Spark Streaming.

Query and analysis

See Overview of real-time query and analysis:

• Query logs in the Log Service console: See Query logs.
• Query logs by using Log Service SDKs/APIs: Log Service provides RESTful APIs that are implemented based on HTTP protocol. The Log Service APIs also provide a full-featured log query API. For more information, see Log Service APIs.

Shipping and storage

• Ship logs to OSS: Store logs for a long term or use E-MapReduce to analyze logs.
• Ship logs to Table Store: Use NoSQL to store logs.

Others

Secure Log Service: Log Service interconnects with cloud security products and uses ISV to consume logs of cloud products.