Log Service provides the real-time consumption feature that allows you to consume data in real time by using SDKs. This topic provides an overview of the real-time consumption feature, including the concept, benefits, scenarios, billing, and consumers.

Real-time consumption

Real-time consumption refers to the real-time consumption of log data by entities such as third-party software, applications in various programming languages, cloud services, and stream computing frameworks. The entities use SDKs to consume the log data in Log Service in real time. This feature reads and writes full data in the first in, first out (FIFO) order. This feature works in a similiar manner as Apache Kafka.

Real-time consumption
Note Both the real-time consumption feature and the query and analysis feature are used to read data. For more information about the difference between the two features, see What are the differences between LogHub and LogSearch?

Scenarios

Real-time consumption is suitable for scenarios such as stream computing and real-time computing. Real-time consumption is time-sensitive and enables data consumption in seconds. You can configure a custom data retention period.

Benefits

Real-time consumption provides the following benefits:
  • Centralized data storage

    Log Service centrally stores log data that is collected from different machines. This way, you can consume the collected data in real time by using SDKs.

  • Data classification and management

    Log Service supports data classification and management. This allows different applications and services to consume data of different types in different projects in real time.

Billing

You are charged for real-time consumption based on multiple billable items, such as read and write traffic and requests. For more information, see Billable items.

Consumers

The following table describes the consumers that are supported by real-time consumption.

Type Consumer Description
Third-party software Splunk You can use Splunk to consume data collected by Log Service in real time. Fore more information, see Ship data to Splunk by using the Splunk add-on for Log Service.
Flume You can use Flume to consume data collected by Log Service in real time. Fore more information, see Use Flume to consume log data.
Logstash You can use Logstash to consume data collected by Log Service in real time. Fore more information, see Use Logstash to consume log data.
QRadar Security information and event management (SIEM) systems, such as IBM QRadar, can consume data collected by Log Service in real time over HTTPS or Syslog. For more information, see Ship logs to a SIEM system over HTTPS and Ship logs to a SIEM system over Syslog.
Applications in various programming languages Applications in various programming languages Applications developed in programming languages such as Java, Python, and Go can consume data collected by Log Service as consumers or consumer groups. For more information, see Consume log data and Use consumer groups to consume log data.
Stream computing Storm You can use the stream computing framework Storm to consume data collected by Log Service in real time. Fore more information, see Use Storm to consume log data.
Flink You can use the stream computing framework Flink to consume data collected by Log Service in real time. Fore more information, see Use open source Flink to consume log data.
Spark You can use the stream computing framework Spark to consume data collected by Log Service in real time. Fore more information, see Use Spark Streaming to consume log data.
Cloud services Function Compute You can use Function Compute to consume data collected by Log Service in real time. Fore more information, see Use Function Compute to consume log data.
Blink You can use Realtime Compute to consume data collected by Log Service in real time. Fore more information, see Use Realtime Compute to consume log data.