After LogHub collects logs, Log Service consumes these logs in the following ways.
|Method||Scenario||Real-time performance||Storage duration|
|Real-time consumption (LogHub)||StreamCompute and Realtime Compute||Real time||Custom|
|Index and query (LogSearch)||Applicable to online query of recent hot data||Real time (delayed for one second in 99.99% cases, and up to three seconds)||Custom|
|Shipping and storage (LogShipper)||Applicable to full log storage for offline analysis||Delayed for 5-30 minutes||Depends on the storage system|
- Obtain a cursor based on conditions such as time, Begin, and End.
- Read logs by using the cursor and step, and return the next cursor.
- Move the cursor continuously to consume logs.
Consume logs by using SDKs
Log Service provides SDKs in multiple programming languages such as Java, Python, and Go. These SDKs support log consumption based on API operations. For more information about the SDKs, see Overview.
Consume logs by using consumer groups
Consumer groups are the advanced method Log Service provides for LogHub consumers to consume logs. Consumer groups provide a lightweight computing framework that allows multiple consumers to concurrently consume data in a Logstore. Consumer groups can also automatically assign shards, maintain the order of log processing, and resume transmission from a breakpoint. Go, Python, and Java SDKs support consumer groups.
Consume logs by using open-source services
Use Flume to consume LogHub logs: you can use Flume to consume logs and import logs to Hadoop file system (HDFS) instances.
Query and analysis
Shipping and storage
Other method for log consumption
Security log service: Log Service connects to cloud security services and uses an independent software vendor (ISV) to consume logs of cloud services.