Simple Log Service provides a real-time consumption feature that allows you to consume data in real time by using SDKs. This topic describes the feature's concept, benefits, use cases, billing rules, and consumers.
Real-time consumption
Real-time consumption allows consumers, such as third-party software, applications in various programming languages, cloud services, and stream computing frameworks, to consume data from Simple Log Service in real time by using SDKs. This feature provides sequential read access to the entire dataset, similar to a message broker like Kafka. You can also use an SPL statement to process data during consumption, and Simple Log Service returns the results. For more information, see Consume logs based on rules.
Both real-time consumption and query and analysis read data. For more information about the differences, see Differences between log consumption and query.
Use cases
Real-time consumption is ideal for scenarios such as stream computing and real-time analytics. It provides low-latency data consumption, typically within seconds. You can also configure a custom data retention period.
Benefits
Real-time consumption provides the following benefits:
-
Centralized data
Simple Log Service centralizes data from different machines, allowing you to consume the collected data in real time by using SDKs.
-
Categorized management
You can use the data categorization features of Simple Log Service to allow different applications and cloud services to consume different types of data from different projects in real time.
Billing
-
If your Logstore uses the pay-by-ingested-data billing mode, Simple Log Service does not charge for real-time consumption. However, if you pull data from a public endpoint of Simple Log Service, you are charged for read traffic over the Internet based on the compressed data size. For more information, see Billable items for the pay-by-ingested-data billing mode. To learn how to view the billing mode of a Logstore, see Manage a Logstore.
-
If your Logstore uses the pay-by-feature billing mode, real-time consumption involves multiple billable items, including read/write traffic and requests. For more information, see Billable items for the pay-by-feature billing mode.
Consumers
The following table describes the consumers that Simple Log Service supports for real-time consumption.
|
Type |
Consumer |
Description |
|
Third-party software |
Flume |
Use Flume to consume data from Simple Log Service in real time. For more information, see Consume data by using Flume. |
|
Logstash |
Use Logstash to consume data from Simple Log Service in real time. For more information, see Consume data by using Logstash. |
|
|
QRadar |
A security information and event management (SIEM) system, such as QRadar, can consume data from Simple Log Service in real time over HTTPS or Syslog. For more information, see Ship logs to a SIEM system and Ship logs to a SIEM system over Syslog. |
|
|
Applications in various programming languages |
Applications (SLS consumption protocol) |
Applications developed in languages such as Java, Python, and Go can consume data from Simple Log Service as a consumer or a consumer group. For more information, see Consume data by using an API and Consume data by using a consumer group. |
|
Stream computing frameworks |
Flink |
Use the stream computing framework Flink to consume data from Simple Log Service in real time. For more information, see Consume data by using Flink. |
|
Spark |
Use the stream computing framework Spark to consume data from Simple Log Service in real time. For more information, see Consume data by using Spark Streaming. |
|
|
Cloud services |
Function Compute |
Use Function Compute to consume data from Simple Log Service in real time. For more information, see Consume log data by using Function Compute. |
|
Blink |
Use Realtime Compute for Apache Flink to consume data from Simple Log Service in real time. For more information, see Consume data by using Realtime Compute for Apache Flink. |