This topic describes typical scenarios in which you can use Log Service for your business. The scenarios include data collection and consumption, data extract, transform, and load (ETL) and stream processing, integration with data warehouses, and real-time query and analysis.

Data collection and consumption

You can use the LogHub module of Log Service to collect large amounts of log data in real time. The log data can be metrics, events, binary logs, text logs, and clickstream data.

Benefits:

  • Ease of use: Log Service provides more than 50 data collection methods to help you build platforms. Log Service also delivers powerful configuration and management capabilities to reduce your O&M workloads.
  • Elastic scalability: Log Service can handle traffic spikes and business growth.
Figure 1. Data collection and consumption

ETL and stream processing

LogHub can connect to multiple stream processing engines and services. LogHub can also monitor the processing progress and generate alerts. You can also SDKs or API operations to consume data based on your business requirements.

  • Ease of use: Log Service provides comprehensive SDKs and programming frameworks for seamless integration with multiple stream processing engines.
  • Monitoring and alerting: Log Service provides comprehensive metrics and an alerting mechanism upon latency.
  • Elastic scalability: Log Service supports auto scaling to process petabytes of data without latency.
Figure 2. ETL and stream processing

Integration with data warehouses

The LogShipper module of Log Service can ship LogHub data to storage services. During the shipping, you can compress the data, define custom partition formats, and specify row or column store.

  • Large data capacity: An unlimited amount of data can be shipped to storage services.
  • Multiple formats: Various storage formats such as row store, column store, and text files are supported.
  • Flexible configurations: Different configurations are supported, which allows you to define custom partition formats.
Figure 3. Integration with data warehouses

Real-time query and analysis

The LogAnalytics module allows you to index LogHub data in real time and query data by using keywords, fuzzy match, contextual query, or SQL aggregate functions. You can also query data within a specified range.

  • Timeliness: You can perform real-time query after data is written to LogHub.
  • High efficiency at low costs: You can index petabytes of data per day. Costs are 85% lower compared with self-managed systems.
  • Strong analysis: Multiple query methods and SQL aggregate functions are supported. Log Service can also generate visualized reports and alerts.
Figure 4. Real-time query and analysis