Realtime Compute for Apache Flink:Large-state jobs

What is State

Apache Flink is an open source framework for real-time stream processing. State in Flink refers to information maintained across multiple events, similar to how memory works in a computer. State allows operators to track historical data when processing unbounded streams. State can be stored as key-value pairs or more complex data types such as lists, arrays, and custom objects. Accessing and updating state data is essential to complex stream processing.

State management in the Realtime Compute for Apache Flink development console automatically manages deployment checkpoints and minimizes storage usage without affecting availability. It also enables efficient savepoint management and sharing between deployments, which is valuable for A/B testing and active/standby failover of large-state deployments.

Issues caused by large state size

Large-state deployments are challenging to maintain. As state size grows, several performance issues emerge.

• Backpressure and performance decrease

  Backpressure occurs when I/O overhead reaches a critical level, increasing processing latency and decreasing throughput measured in transactions per second (TPS).

• Low resource utilization

  Stateful operators typically have a large amount of idle CPU resources. The larger the state, the more significant the resource waste.

• Checkpoint and savepoint timeout

  Large state sizes increase the likelihood of checkpoint and savepoint timeouts, which lengthens recovery time after restarts and increases end-to-end exactly-once processing latency.

• Slow startup and scaling

  During startup or scaling, each operator must recover its local data from the full deployment state. Recovery time is proportional to state size, which increases service interruption for large-state deployments.

Workflow of tuning large-state deployments

To prevent these issues, follow these steps to optimize state management for large-state deployments:

1. Identify potential bottlenecks

   Use diagnostic tools to assess the current deployment status and determine whether performance bottlenecks are caused by improper state management. For more information, see Deployment performance.

2. Use the latest engine version

   Realtime Compute for Apache Flink continuously optimizes the state module of the Ververica Runtime (VVR) engine, and the latest version typically delivers the best performance. VVR is an enterprise-grade engine fully compatible with Apache Flink, equipped with GeminiStateBackend to improve state access, checkpoints, and state recovery. GeminiStateBackend automatically tunes parameters for optimal performance, eliminating the need for manual configuration. Before applying other optimizations, make sure you use the latest VVR version. For more information, see GeminiStateBackend, Configurations of GeminiStateBackend, and Job cloning and engine version upgrade.

3. Apply different tuning strategies for different issues

   • Runtime performance decrease (Backpressure)

     Optimize SQL queries, adjust time-to-live (TTL) to reduce state data, and then tune memory allocation and parallelism settings to reduce disk I/O. For more information, see Control state size to reduce backpressure in SQL deployments and Control state size to reduce backpressure using the Datastream API.

   • Checkpoint or savepoint timeout

     Optimize runtime performance to reduce backpressure, improve synchronous-phase performance, adjust parallelism to reduce per-subtask state size, and consider using native-format savepoints. For more information, see Diagnose and tune checkpoint and savepoint timeouts.

   • Slow startup and scaling

     If local disk space is sufficient, enable local recovery. You can also use the lazy loading and delayed pruning features of GeminiStateBackend to speed up startup and scaling. For more information, see Improve startup and scaling speed.