After LogHub collects logs, Log Service consumes these logs in the following ways.

Method Scenario Real-time performance Storage duration
Real-time consumption (LogHub) StreamCompute and Realtime Compute Real time Custom
Index and query (LogSearch) Applicable to online query of recent hot data Real time (delayed for one second in 99.99% cases, and up to three seconds) Custom
Shipping and storage (LogShipper) Applicable to full log storage for offline analysis Delayed for 5-30 minutes Depends on the storage system

Real-time consumption

LogHub provides the API operation to pull logs and support real-time log consumption. Log Service consumes logs in a shard in the following steps:
  1. Obtain a cursor based on conditions such as time, Begin, and End.
  2. Read logs by using the cursor and step, and return the next cursor.
  3. Move the cursor continuously to consume logs.
Note To consume or query logs, Log Service needs to read logs. For more information about the differences between consuming logs and querying logs, see Differences between log consumption and log query.

Consume logs by using SDKs

Log Service provides SDKs in multiple programming languages such as Java, Python, and Go. These SDKs support log consumption based on API operations. For more information about the SDKs, see Overview.

Consume logs by using consumer groups

Consumer groups are the advanced method Log Service provides for LogHub consumers to consume logs. Consumer groups provide a lightweight computing framework that allows multiple consumers to concurrently consume data in a Logstore. Consumer groups can also automatically assign shards, maintain the order of log processing, and resume transmission from a breakpoint. Go, Python, and Java SDKs support consumer groups.

Consume logs by using StreamCompute
Consume logs by using cloud services

Consume logs by using open-source services

Use Flume to consume LogHub logs: you can use Flume to consume logs and import logs to Hadoop file system (HDFS) instances.

Query and analysis

For more information, see Overview. You can query and analyze logs in the following ways:
  • Query logs in the Log Service console. For more information, see Overview.
  • Query logs by using the SDKs or API operations of Log Service. Log Service provides HTTP-based RESTful API operations. The API operations support full-featured log queries. For more information, see Overview.

Shipping and storage

Other method for log consumption

Security log service: Log Service connects to cloud security services and uses an independent software vendor (ISV) to consume logs of cloud services.