This tutorial guides you through the completion of real-time R&D and visual analytics in Dataphin using Ververica Flink, providing a foundational understanding of Dataphin's real-time R&D process.
Background information
A company's order system, which utilizes MySQL for its storage database, contains the 'oms_order' table for storing orders. To support a marketing campaign, there is a need to tally the sales quantity of each product type in real-time, enabling swift adjustments to the campaign strategy.
Below is a schematic representation of the data flow:
Ververica Flink tasks process data from the order system's MySQL in real-time, summarizing sales quantities by product type and updating another MySQL database for OLAP analysis.
BI tools, such as Quick BI, access the MySQL data for visual analytics.
Process guide
The basic process for real-time R&D using Dataphin is outlined below:
Main Process | Description | Operation Guide |
Preparations | Prior to engaging in real-time R&D with Dataphin, you must prepare related cloud resources. This includes setting up an Alibaba Cloud account, activating and configuring Dataphin and AccessKey, activating and configuring Ververica Flink computing resources, activating Quick BI (optional), and preparing the data source. | |
Project and Computing Source | A real-time project serves as the fundamental unit for developing real-time tasks and can be associated with Ververica Flink computing resources for real-time R&D. | |
Real-time Meta Table | Meta tables facilitate cross-storage type data management. Input, output, and dimension tables required for development can be created and managed through meta tables. | |
Real-time Task | Create a Flink_SQL stream task to read or write data from the data source as mapped by the meta table. | |
Task Operation and Maintenance | Once real-time R&D is complete, submit the task to the Operation Center. Here, you can start, stop, view logs, monitor task metrics, configure alerts, and perform other operational tasks. | |
Visual Analytics | Optional. Once the real-time instance is initiated, it continuously reads data from the source table, processes and computes it using Ververica Flink, and then writes the results to the output table in real-time. For visual analytics of the data in the output table, BI tools like Quick BI can be utilized. |