SLS enables real-time data consumption through SDKs, providing sequential read access to log data similar to Kafka.
Real-time consumption
Real-time consumption allows third-party software, multi-language applications, cloud services, and stream computing frameworks to consume SLS data in real time through SDKs. It provides sequential read access to the full dataset, similar to Kafka. You can use SPL statements to process data during consumption, and SLS returns the results. Consume logs based on rules.
Real-time consumption and query-analysis both read data but serve different purposes. Differences between log consumption and query.
Use cases
Real-time consumption suits stream computing and real-time analytics scenarios with low-latency data delivery, typically within seconds. You can configure a custom data retention period.
Benefits
Benefits of real-time consumption:
-
Centralized data
SLS centralizes data from multiple machines for real-time SDK-based consumption.
-
Categorized management
SLS data categorization lets different applications and cloud services consume specific data types from separate projects.
Billing
-
If your Logstore uses pay-by-ingested-data billing, real-time consumption is free. However, pulling data from a public SLS endpoint incurs Internet read traffic charges based on compressed data size. Billable items for the pay-by-ingested-data billing mode. Manage a Logstore.
-
If your Logstore uses pay-by-feature billing, real-time consumption incurs charges for read/write traffic and requests. Billable items for the pay-by-feature billing mode.
Consumers
SLS supports the following consumers for real-time consumption.
|
Type |
Consumer |
Description |
|
Third-party software |
Flume |
Consume SLS data in real time with Flume. Consume data by using Flume. |
|
Logstash |
Consume SLS data in real time with Logstash. Consume data by using Logstash. |
|
|
QRadar |
SIEM systems such as QRadar can consume SLS data in real time over HTTPS or Syslog. Ship logs to a SIEM system. Ship logs to a SIEM system over Syslog. |
|
|
Applications in various programming languages |
Applications (SLS consumption protocol) |
Java, Python, and Go applications can consume SLS data as individual consumers or consumer groups. Consume data by using an API. Consume data by using a consumer group. |
|
Stream computing frameworks |
Flink |
Consume SLS data in real time with Flink. Consume data by using Flink. |
|
Spark |
Consume SLS data in real time with Spark Streaming. Consume data by using Spark Streaming. |
|
|
Cloud services |
Function Compute |
Consume SLS data in real time with Function Compute. Consume log data by using Function Compute. |
|
Blink |
Consume SLS data in real time with Realtime Compute for Apache Flink. Consume data by using Realtime Compute for Apache Flink. |