This topic describes common scenarios for Realtime Compute for Apache Flink in real-time big data processing, categorized by business department and technical field.
Background information
As a stream computing engine, Flink is widely used for real-time data processing. Examples include online service logs from Elastic Compute Service (ECS) instances and sensor data from Internet of Things (IoT) scenarios. Flink can also subscribe to binary logging (binlog) updates from relational databases, such as ApsaraDB RDS (RDS) and PolarDB. You can then use services such as DataHub, Simple Log Service (SLS), and Kafka to collect this real-time data into Realtime Compute for Apache Flink for analysis and processing. Finally, you can write the analysis results to various data services. Examples include MaxCompute, Hologres for interactive analysis, Platform for AI, and Elasticsearch. This improves data utilization and helps meet business requirements.
Department scenarios
From a business perspective, Realtime Compute for Apache Flink is used in the following scenarios:
-
Business departments: real-time risk control, real-time recommendations, and real-time index building for search engines.
-
Data departments: real-time data warehousing, real-time reports, and real-time dashboards.
-
O&M departments: real-time monitoring, real-time anomaly detection and alerting, and end-to-end debugging.
Technical fields
From a technical perspective, Realtime Compute for Apache Flink is used in the following scenarios:
Real-time ETL and data streams
Real-time extract, transform, and load (ETL) and data streams deliver data from one point to another in real time. This process can include data cleaning and integration tasks. Examples include building a search index in real time or running ETL processes for a real-time data warehouse.
Real-time data analytics
Data analytics is the process of extracting and integrating information from raw data to meet business goals. For example, you can view the top 10 best-selling products each day, the average warehouse turnover time, the average document click rate, and the push notification open rate. Real-time data analytics performs this process in real time, with results often displayed on real-time reports or dashboards.
Event-driven applications
An event-driven application processes or responds to a series of subscribed events. These applications often rely on internal state. Examples include fraud detection, risk control systems, and O&M anomaly detection systems. When a user's behavior triggers a risk control rule, the system catches the event. It then analyzes the user's current and past behavior to decide whether to apply risk control measures.
Risk control system
Realtime Compute for Apache Flink can process complex stream and batch tasks. It provides powerful APIs to perform complex mathematical calculations and run complex event processing rules. This helps businesses analyze data in real time and improves their risk control capabilities. For example, you can detect click behaviors in an app or identify irregular changes in IoT data streams.
The flowcharts for the technical fields described above are from the official Apache Flink website.