Overview - Log consumption and shipping| Alibaba Cloud Documentation Center

After LogHub collects logs, Log Service consumes these logs in the following ways.


Method	Scenario	Real-time performance	Storage duration
Real-time consumption (LogHub)	StreamCompute and Realtime Compute	Real time	Custom
Index and query (LogSearch)	Applicable to online query of recent hot data	Real time (delayed for one second in 99.99% cases, and up to three seconds)	Custom
Shipping and storage (LogShipper)	Applicable to full log storage for offline analysis	Delayed for 5-30 minutes	Depends on the storage system

Real-time consumption

LogHub provides the API operation to pull logs and support real-time log consumption. Log Service consumes logs in a shard in the following steps:

Obtain a cursor based on conditions such as time, Begin, and End.
Read logs by using the cursor and step, and return the next cursor.
Move the cursor continuously to consume logs.

Note To consume or query logs, Log Service needs to read logs. For more information about the differences between consuming logs and querying logs, see Differences between log consumption and log query.

Consume logs by using SDKs

Log Service provides SDKs in multiple programming languages such as Java, Python, and Go. These SDKs support log consumption based on API operations. For more information about the SDKs, see Overview.

Consume logs by using consumer groups

Consumer groups are the advanced method Log Service provides for LogHub consumers to consume logs. Consumer groups provide a lightweight computing framework that allows multiple consumers to concurrently consume data in a Logstore. Consumer groups can also automatically assign shards, maintain the order of log processing, and resume transmission from a breakpoint. Go, Python, and Java SDKs support consumer groups.

Consume logs by using StreamCompute

Use the Spark Streaming client to consume logs.
Use Storm Spout to consume logs.
Use the Flink connector to consume logs. The Flink connector consists of the consumer and producer.

Consume logs by using cloud services

Use CloudMonitor to consume logs in monitoring scenarios.
Use Function Compute to consume logs.
Use E-MapReduce to consume logs. For more information, see Use Storm to consume LogHub logs and Use Spark Streaming to consume LogHub logs.

Consume logs by using open-source services

Use Flume to consume LogHub logs: you can use Flume to consume logs and import logs to Hadoop file system (HDFS) instances.

Query and analysis

For more information, see Overview. You can query and analyze logs in the following ways:

Query logs in the Log Service console. For more information, see Overview.
Query logs by using the SDKs or API operations of Log Service. Log Service provides HTTP-based RESTful API operations. The API operations support full-featured log queries. For more information, see Overview.

Shipping and storage

Ship logs to Object Storage Service (OSS): stores logs for a long period or analyzes logs based on E-MapReduce.
Use Function Compute to customize shipping.
Ship data to MaxCompute by using DataWorks: ships logs to MaxCompute by using Data Integration of DataWorks to perform big data analysis.

Other method for log consumption

Security log service: Log Service connects to cloud security services and uses an independent software vendor (ISV) to consume logs of cloud services.