Open source Apache Flink performance comparison

Performance comparison between real-time computing Flink version and open source Apache Flink

The real-time computing Flink version product is a natural cloud-native infrastructure.

On the core computing engine, compared with the open source Apache Flink Alibaba Cloud, several core functions have been optimized, and these optimizations have also passed the tempering of Alibaba's internal business. At present, real-time computing Flink products support the real-time data services of nearly 100 business divisions of the Ali Group. Through a large number of business practices, the product has been debugged to the best effect in terms of supporting storage, scheduling, and network transmission.

In terms of plug-ins, the product has dozens of enhanced connectors built in, which can be connected to all mainstream open source data storage, including MySQL, HBase, HDFS, Alibaba Cloud SLS, etc., which are naturally integrated and ready to use out of the box. In terms of development platform, it provides an enterprise-level one-stop development platform with built-in development and operation and maintenance capabilities, eliminating the trouble of self-build and improving the overall experience of enterprise users.

The real-time computing Flink version supports multi-language development environments such as SQL, Java, and Python, provides full lifecycle management of development tasks, supports enterprise-level security mechanisms based on OIDC and RBAC, and has full-link monitoring and alarming based on the Prometheus protocol. It provides its own AutoPilot intelligent tuning system, which intelligently helps users to tune the parameters of Flink tasks, including resource tuning and concurrency tuning. The product can completely adapt to the traffic of the business without any manual debugging (intelligent tuning is the core advantage of the real-time computing Flink version of the product).

To develop

In terms of development, it has rich data connection capabilities and a one-stop multi-language development environment. It has a variety of built-in function libraries, which is convenient for users to debug code, and can also carry out multi-tenant development, task debugging, and test simulation.

Operation and maintenance

In terms of operation and maintenance, it supports monitoring and alarming of the entire link, and automatic alarms can be issued for data delays, data anomalies, and service interruptions that occur during users' use. In terms of intelligent operation and maintenance, it supports automatic intelligent diagnosis and optimization, and can automatically help users perform performance tuning, job tuning, parameter tuning, and resource tuning based on business traffic, and can diagnose and optimize problems.

At the resource level, on the basis of open source, finer-grained and more refined resource allocation is achieved, so that each job and each operator can be configured at the granularity of CPU and memory, which greatly optimizes resource utilization and helps users Save costs, improve service stability, and reduce the probability of OM.

With the original factory's operation and maintenance service, SLA99.9% guarantee, full link fault tolerance and system stability guarantee, it fully solves the worries of users.

Cost

In terms of cost, through cost optimization on the cloud, the overall TCO of users can be reduced while performance is improved, which is also an advantage of core performance.

In the standard test of stream computing based on Nexmark, the product performance of the real-time computing Flink version is about three times that of the open source. Relying on the practical optimization accumulated by the strong R&D team of Alibaba Group in internal core business scenarios, the product can reduce the basic cost of users First, highlight the core advantages.

The real-time computing Flink version also has cloud-native elastic expansion capabilities, which can help users save resources reasonably and improve resource utilization. The product payment type supports yearly and monthly payment, and also supports pay-as-you-go, which better adapts to different needs.

Safety

The security layer improves user experience through containerized task isolation, and supports multiple requirements such as tenant isolation, security isolation, and VPC isolation. At the same time, it is directly connected with Alibaba's account system. Users can seamlessly control the security between products based on Alibaba Cloud's account. It also supports open identity authentication protocols such as role-based and OIDC, which greatly improves business security.

On the whole, the enterprise version has better functionality and stability than the open source version. In addition to the advantages in operation and maintenance, the out-of-the-box operation also makes users more convenient.