This topic describes the development history of Realtime Compute.
Alibaba Cloud Realtime Compute offers an end-to-end solution of stream processing based on Flink, from job development to administration. Based on many years of experience of Alibaba Group in big data technologies and business scenarios, Realtime Compute allows you to take advantage of the powerful capabilities of advanced computing engines. By leveraging the experience and expertise of Alibaba Group in streaming data services, you can easily and quickly utilize the benefits of stream processing to accelerate the growth of big data services.
- Beginning: Real-time big screen service of the Double 11
Realtime Compute has its beginnings in the big screen service of the Double 11. With years of experience and development, the small team that once provided the real-time big screen service and limited real-time reporting services has become an independent and reliable cloud computing team. Realtime Compute provides an end-to-end cloud solution of stream processing based on years of experience in real-time computing products, architecture, and business scenarios. We strive to offer powerful support for small and medium-sized enterprises (SMEs) in terms of real-time big data processing.
- Early stage: Development based on open source Flink
Alibaba Group adopted open source Flink to support the big screen service during the Double 11. Flink code was created for stream processing. During the early stages, stream processing services were provided on a small scale. Developers used Flink APIs to create stream processing jobs. Therefore, developers must have proficient technical skills, handle debugging challenges, and perform large amounts of repetitive tasks.
- Continuous optimization: Development based on Flink APIs
To handle large amounts of repetitive work, Alibaba Group engineers started working on data encapsulation and abstraction. Based on Flink APIs, they developed a large number of reusable components for data statistics, such as the basic programming components for simple filtering, aggregation, and windows. Based on these components, an XML description language is provided. With this design, Realtime Compute users can use Extensible Markup Language (XML) to describe and integrate Flink components, and create end-to-end real-time computing processes. This programming method eliminates large amounts of repetitive development work that is required at the underlying layer, and reduces the requirements for development skills. This programming method is different from the SQL method that is most familiar to data analysts. Therefore, analysts must learn more about the programming components and XML syntax.
- Maturity: Flink SQL development
Any emerging technology is only adopted by a small group in the beginning. With the growth of this technology and the reduction in adoption costs, it will be widely accepted. Therefore, Alibaba Cloud engineers are working to enable stream processing technologies to be widely adopted by improving the technology and decreasing adoption costs. Thanks to years of experience in relational databases, Alibaba Group engineers developed Flink SQL to replace the programming method that is based on XML and Flink components. Flink SQL allows you to write SQL code for real-time computing and data processing. All these improvements are integrated into Flink, the core computing engine of Realtime Compute. For this computing engine, a single cluster includes up to thousands of machines. An average of hundreds of billions of messages can be processed per day, and the amount of data that is processed per day nearly reaches the PB level. Flink clusters have become the core stream processing clusters of Alibaba Group.Flink SQL offers the following benefits:
- Flink SQL supports a wide range of SQL functions, which improves the technical maturity of users.
- You can use familiar SQL models for easy adoption of Realtime Compute.