Features and benefits - Realtime Compute for Apache Flink

This topic describes the features and benefits of fully managed Flink of Realtime Compute for Apache Flink and compares this Alibaba Cloud service with Apache Flink.

Category	Item	Description	Benefit
Performance and cost	Core performance	GeminiStateBackend is a backend storage system developed by Alibaba Cloud. It is fully compatible with APIs and the Apache Flink community and adopts a new architecture and data structure design. GeminiStateBackend supports compute-storage separation, which frees you from the limit on the local disk storage of state data. It also supports key-value separation, which significantly improves the efficiency of deployments that involve dual-stream or multi-stream JOIN. GeminiStateBackend also supports adaptive parameter tuning, which eliminates manual parameter adjustment. The Nexmark benchmark test results show that GeminiStateBackend provides twice the stream computing performance of Apache Flink. For more information, see GeminiStateBackend and Performance white paper (Nexmark performance testing). The SQL engine is compatible with the syntax of Apache Flink and provides optimizations that include but not limited to the state structure optimization of operators, latency materialization at the computing layer, Codegen enhancement, and optimization for scenarios in which JOIN operations are performed, such as the cache enhancement of dimension tables, optimization for data skews, miniBatch optimization for JOIN operations on streams, and fine-grained state settings. This helps improve CPU utilization and memory usage and reduce state storage usage.	Fully managed Flink provides better engine performance and finer-grained resource allocation, which reduces the total cost of ownership (TCO) compared with that of Apache Flink. Fully managed Flink also supports multiple billing methods and intelligent auto scaling. This helps you use resources in a more fine-grained manner.
	Resource utilization	Allows you to automatically scale your clusters in response to business loads.
		Supports the Autopilot feature. After this feature is enabled, the system can automatically monitor and adjust deployment resources and apply different resource plans for different scenarios. This helps you scale to meet the requirements of business peaks while keeping costs within a reasonable range.
		Allows you to manage resources in a fine-grained manner and supports fine-grained resource configurations (CPU cores and memory) at the SQL operator level. This improves the resource utilization of large-scale deployments by 100%.
	Billing method	Provides the subscription and pay-as-you-go billing methods, which you can choose based on your business requirements.
Featured capabilities	Real-time data ingestion into data lakes or data warehouses	Supports real-time synchronization of data from a database, real-time synchronization of data from tables in shared databases, and real-time synchronization of schema changes.	This feature allows you to ingest data in business databases or message-oriented middleware that uses table sharding into data lakes or data warehouses in real time.
	Real-time fraud detection	Supports enterprise-class complex event processing (CEP), and allows you to dynamically configure rules for a deployment without having to restart the deployment. This delivers uninterrupted production-level capabilities in mission-critical scenarios, such as online real-time fraud detection.	This feature is suitable for mission-critical scenarios, such as real-time marketing, real-time fraud detection, Threat Detection Service (TDS), to improve development efficiency and large-scale data processing capabilities, while ensuring business continuity.
	Upstream and downstream connectors	Supports more than 30 mainstream engines of Alibaba Cloud services and the Apache Flink community. The engines are used in various upstream and downstream data stores, such as databases, message-oriented middleware, data warehouses, lake formats, and file systems. Supports the Faker connector to generate data that closely resembles actual business data for testing. Improves the usability and stability of the connectors compared with the connectors provided by Apache Flink. Allows you to create custom connectors to access various external storage systems from fully managed Flink.	You do not need to develop or connect to various upstream and downstream ecosystems. The connectors help ensure system stability and performance.
Development efficiency	Develop a draft	Programming languages: Fully managed Flink provides an end-to-end development and management platform, which supports various programming languages such as SQL, Java, Scala, and Python.	You do not need to build a development environment or connect to Apache Flink. Flink SQL is easy to use in the overall development environment.
		Multi-version support: Fully managed Flink supports mainstream Flink versions. You can compare deployment code between different versions and perform rollback operations on the code.
		Metadata management: You can create catalogs to connect fully managed Flink to common upstream and downstream components, such as MySQL, Hive, Hologres, Data Lake Formation (DLF), and Kafka. This helps you manage and use metadata in a unified manner.
		User-defined functions (UDFs): You can easily manage and use UDFs.
		Code templates: More than 20 templates that are used in common scenarios of Flink SQL are provided to help you quickly understand how to use Flink SQL to build deployment code.
	Code debugging	Test data management: Fully managed Flink supports online sampling and management of mock testing data to help you build test processes.	Programmers and even data analysts can debug and publish drafts. This significantly reduces the costs for debugging and testing, and reduces the time required to publish drafts.
		Fast deployment and debugging: Fully managed Flink allows you to start or cancel deployments in session clusters within seconds. This makes deployment debugging more efficient.
		Display of intermediate results: Intermediate results can be displayed. This improves the efficiency of debugging complex SQL statements.
		Environment isolation: The development environment is isolated from the production environment. This way, deployments and data in the production environment are not affected during the debugging process.
Deployment operations	Monitoring and alerting	Provides various metrics and aggregate dimensions to help you resolve issues, such as deployment delays, data skew, and backpressure.	These features significantly improve system stability, reduce the O&M workload, and simplify tuning operations. They allow you to manage resources in a fine-grained manner to significantly reduce costs. Furthermore, Alibaba Cloud provides high availability assurances for the service.
	Monitoring and alerting	Sends alerts to recipients over DingTalk or by email, text message, or phone at the earliest opportunity. You can also connect fully managed Flink to your internal unified monitoring and alerting system Prometheus.
	Issue analysis and diagnostics	Allows you to dynamically modify the configuration of a deployment without the need to cancel the deployment. For example, you can change the log level and enable or disable the flame graph without canceling the deployment.
	Issue analysis and diagnostics	Provides intelligent diagnostics for common issues, such as backpressure, deployment exceptions, and TaskManager discontinuity. It is also capable of quickly identifying issues based on logs, and providing suggestions for tuning and modifications. You can also enable Autopilot to automatically locate issues.
	High availability	Delivers Service Level Agreement (SLA)-guaranteed service availability of 99.9% for the maintenance service.
		Supports end-to-end automatic fault recovery and fault-tolerant JobManager to prevent single points of failure. This improves service stability.
		Supports fast single-node fault recovery to balance between data consistency and service continuity.
	Status management	Provides complete system checkpoints and savepoint lifecycle management of deployments, and supports state data compatibility checks and state data migration to maximize the reuse of original state data.
Enterprise security	Isolation	Supports tenant-level and project-level resource isolation and code isolation to meet the data security requirements when different teams collaborate on projects.	These features support the collaboration between multiple departments within an enterprise and meet the internal and external audit requirements of the enterprise.
Enterprise security	Access control	Uses the Alibaba Cloud account system to support access control of multiple roles.