Realtime Compute for Apache Flink:FAQ and solutions for ingestion

Symptom

• The job restarts repeatedly during the snapshot phase.

• The JobManager logs contain an OutOfMemoryError (OOM) stack trace.

• On the Alarm tab, the Num of remaining SnapshotSplits and Num of processed SnapshotSplits metrics show exceptionally high values.

Cause

During the snapshot phase, the MySQL source persists all table shard metadata to the Flink job's state. If the job handles a large volume of data or uses very small shard sizes, the JobManager creates an excessive number of shards, which exhausts its available memory.

Solution

1. Increase the memory resources allocated to the JobManager.

2. Adjust the following parameters to increase the JobManager's heap and off-heap memory:

   • jobmanager.memory.heap.size

   • jobmanager.memory.off-heap.size

Symptom

• The TaskManager runs out of memory late in the snapshot phase, typically when only a small number of shards remain.

• Searching the TaskManager logs for using select statement reveals that the last unbounded query involves a very large volume of data.

Cause

Prolonged data reading during the snapshot phase causes incremental data to accumulate in the final shards. When the TaskManager processes these large accumulated shards, it runs out of memory.

Solution

1. Set the following option:

      scan.incremental.snapshot.unbounded-chunk-first.enabled: true

2. Re-run the snapshot.

Symptom

• The job enters the incremental phase but fails during state restoration.

• The JobManager logs show an OOM.

Cause

VVR 11.1 and earlier versions may not properly clean up persisted table schema information from the job's state after transitioning to the incremental phase. This leftover schema data accumulates and causes an OOM when the job restores from a checkpoint.

Solution

1. Upgrade to VVR 11.2 or later.

Symptom

• The job continues running without restarting after a lock-free table schema change.

• The CurrentFetchTimeLag metric progresses as expected, indicating that data is being fetched.

• The MySQL source stops producing new data and the CurrentEmitTimeLag metric stops updating.

Cause

VVR 11.1 and earlier versions cannot correctly handle DDL events generated by lock-free schema change tools such as pt-osc, which causes the data pipeline to stall.

Solution

1. Upgrade to VVR 11.2 or later.

2. Set the following option:

      scan.parse.online.schema.changes.enabled: true

Symptom

• The job unexpectedly restarts following a lock-free table schema change (for example, using pt-osc).

• The Transform operator logs indicate a column type mismatch error.

Cause

In VVR 11.1, if a significant volume of data is inserted into a table during a lock-free schema change, the engine may generate an unparsable event.

Solution

1. Upgrade to VVR 11.2 or later.

2. Perform a stateful restart from a savepoint that was created before the lock-free schema change.

Symptom

• A stateful restart from a savepoint created before a table schema change fails.

• The error message indicates a table schema mismatch exception while consuming binary logs.

Cause

VVR 11.1 and earlier versions do not support stateful restarts from savepoints that contain an incompatible table schema.

Solution

1. Upgrade to VVR 11.2 or later.

2. After the upgrade, restart the job from a pre-schema-change savepoint.