Realtime Compute for Apache Flink:FAQ and solutions for ingestion

Symptom

• The job restarts repeatedly during the snapshot phase.

• The JobManager logs contain an OutOfMemoryError (OOM) stack trace.

• On the Alarm tab, the Num of remaining SnapshotSplits and Num of processed SnapshotSplits metrics show exceptionally high values.

Cause

During the snapshot phase, the MySQL source persists all table shard metadata to the Flink job's state. If the job handles a large volume of data or uses very small shard sizes, the JobManager creates an excessive number of shards, which exhausts its available memory.

Solution

1. Increase the memory resources allocated to the JobManager.

2. Adjust the following parameters to increase the JobManager's heap and off-heap memory:

   • jobmanager.memory.heap.size

   • jobmanager.memory.off-heap.size

Symptom

• The TaskManager runs out of memory late in the snapshot phase, typically when only a small number of shards remain.

• Searching the TaskManager logs for using select statement reveals that the last unbounded query involves a very large volume of data.

Cause

Prolonged data reading during the snapshot phase causes incremental data to accumulate in the final shards. When the TaskManager processes these large accumulated shards, it runs out of memory.

Solution

1. Set the following option:

      scan.incremental.snapshot.unbounded-chunk-first.enabled: true

2. Re-run the snapshot.

Symptom

• The job enters the incremental phase but fails during state restoration.

• The JobManager logs show an OOM.

Cause

VVR 11.1 and earlier versions may not properly clean up persisted table schema information from the job's state after transitioning to the incremental phase. This leftover schema data accumulates and causes an OOM when the job restores from a checkpoint.

Solution

1. Upgrade to VVR 11.2 or later.

Symptom

• The job continues running without restarting after a lock-free table schema change.

• The CurrentFetchTimeLag metric progresses as expected, indicating that data is being fetched.

• The MySQL source stops producing new data and the CurrentEmitTimeLag metric stops updating.

Cause

VVR 11.1 and earlier versions cannot correctly handle DDL events generated by lock-free schema change tools such as pt-osc, which causes the data pipeline to stall.

Solution

1. Upgrade to VVR 11.2 or later.

2. Set the following option:

      scan.parse.online.schema.changes.enabled: true

Symptom

• The job unexpectedly restarts following a lock-free table schema change (for example, using pt-osc).

• The Transform operator logs indicate a column type mismatch error.

Cause

In VVR 11.1, if a significant volume of data is inserted into a table during a lock-free schema change, the engine may generate an unparsable event.

Solution

1. Upgrade to VVR 11.2 or later.

2. Perform a stateful restart from a savepoint that was created before the lock-free schema change.

Symptom

• A stateful restart from a savepoint created before a table schema change fails.

• The error message indicates a table schema mismatch exception while consuming binary logs.

Cause

VVR 11.1 and earlier versions do not support stateful restarts from savepoints that contain an incompatible table schema.

Solution

1. Upgrade to VVR 11.2 or later.

2. After the upgrade, restart the job from a pre-schema-change savepoint.

Symptom

• The Flink CDC job fails during the snapshot phase with an error indicating that the required binary log position no longer exists or has been purged.

Cause

A Flink CDC job operates in two sequential phases with different binary log consumption patterns:

• Snapshot phase: Reads historical data and replays relatively older binary log entries. If the snapshot phase takes a long time, MySQL may purge the binary log files that the job still needs before the snapshot completes. This causes the job to fail because the required log position is no longer available.

• Incremental phase: Reads the most recent binary log entries in near real time. Because the incremental phase always tracks the latest log position, it is generally not affected by standard binary log purge policies.

Solution

1. Check whether your job is currently in the snapshot phase or the incremental phase by reviewing the job monitoring metrics (for example, the Num of remaining SnapshotSplits metric: a value greater than zero indicates the job is still in the snapshot phase).

2. If the job fails in the snapshot phase due to binary log purge, do one or both of the following:

   • Extend the binary log retention period on your MySQL instance to cover the expected snapshot duration:

     • MySQL 5.7 and earlier: set expire_logs_days to a sufficient number of days.

     • MySQL 8.0 and later: set binlog_expire_logs_seconds to a sufficient number of seconds.

   • Reduce snapshot duration by performing a stateless restart and increasing the job parallelism. A higher degree of parallelism speeds up the snapshot phase, reducing the window during which binary logs can be purged.