Connector Behavior During Upstream and Downstream Scaling - Realtime Compute for Apache Flink

When an upstream or downstream system scales — for example, when Kafka partitions are added, a database instance restarts, or compute resources change — each Flink connector responds differently. Some connectors adapt automatically; others require manual intervention to avoid data loss or uneven distribution. Use this reference to determine whether a job recovers on its own or requires action from you.

How failover works

When an external system becomes temporarily unavailable, connectors first attempt to reconnect. If reconnection succeeds, the job continues without a failover. If all reconnection attempts fail — typically due to prolonged unavailability — the connector throws an exception, causing task failure. Realtime Compute for Apache Flink then performs a failover, re-orchestrating tasks and attempting to reconnect to the external system.

Job faults such as task failures or vertex downtime trigger an automatic failover. The goal is to restore the job to a normal state and guarantee accurate, consistent data processing.

Connector behavior and recommended actions

Connector	Behavior during scaling	Recommended action	Writing to sink dependent on checkpointing?
Message Queue for Kafka	Realtime Compute for Apache Flink dynamically detects new partitions. If the new partition count is not an integer multiple of the current parallelism, data will not be distributed evenly across subtasks, which may cause some subtasks to be idle and others to be overloaded.	After adding partitions, adjust the job parallelism to a value that evenly divides the new partition count. For example, if the partition count increases from three to eight, set the parallelism to four or eight.	Yes, for exactly-once delivery
Upsert Kafka	Same as Message Queue for Kafka.	Same as Message Queue for Kafka.
Hologres	During instance scaling or restart, connections may be disrupted. Realtime Compute for Apache Flink tries to re-establish the connection until timeout, then performs a failover until the Hologres instance is back online.	Perform a stateless startup. Realtime Compute for Apache Flink reads Hologres tables by table name, so no state restoration is required.	No
Simple Log Service	VVR 8.0.8 or earlier: Realtime Compute for Apache Flink fails over the job to adapt to partition changes. VVR 8.0.9 or later: If `enableNewSource` is set to `true`, no failover is triggered. If `shardDiscoveryIntervalMs` is also set, partition count changes are detected at a regular interval.	After a failover, the job adapts to the new partition count automatically. To avoid failover, manually restart the job after changing the partition count.	No
MySQL	Connection disruptions caused by instance scaling or restart are detected automatically. Realtime Compute for Apache Flink restarts the affected task and attempts to recreate the connection. If the database endpoint is unchanged and the service is available, the connector recovers the job without a failover. If reconnection fails for a prolonged period, a failover is triggered. After recovery, tasks are re-orchestrated and the connector attempts to reconnect. Note Primary-secondary switchover or cluster restart can cause temporary connection disruption. If the connection is not restored within a reasonable time, a failover is triggered. To avoid this, cancel the job before making configuration changes and restart it when the changes are complete.	Assess the impact of a job restart before scaling. If the endpoint changes: update your code, redeploy the program, and start the job. If the endpoint does not change: no restart is needed.	No
ApsaraDB RDS for MySQL	Same as MySQL.	Same as MySQL.	No
JDBC	Same as MySQL.	Same as MySQL.	No
AnalyticDB for PostgreSQL	Same as MySQL.	Same as MySQL.	No
AnalyticDB for MySQL V3.0	Same as MySQL.	Same as MySQL.	No
Time series database: InfluxDB	Same as MySQL.	Same as MySQL.	No
OceanBase	Same as MySQL.	Same as MySQL.	No
PolarDB for PostgreSQL	Same as MySQL.	Same as MySQL.	No
Lindorm	Same as MySQL.	Same as MySQL.	No
ApsaraDB for HBase	Same as MySQL.	Same as MySQL.	No
PostgreSQL CDC	N/A	N/A	N/A
Elasticsearch	Same as MySQL.	Same as MySQL.	Yes
StarRocks	Same as MySQL.	Same as MySQL.	Yes, for exactly-once delivery
MaxCompute	After a scale-down, if MaxCompute lacks sufficient compute resources to read from or write to Realtime Compute for Apache Flink at the current job parallelism, affected subtasks throw errors until resources become available.	Before scaling down, carefully evaluate data traffic. Alternatively, reduce the job parallelism to match available resources.	Batch Tunnel mode: Yes
DataHub	The connector cannot automatically detect partition count changes.	Manually restart the job after changing the partition count so it can adapt to the new configuration.	No
Tair	Standard master-replica instances: Support imperceptible scaling, including adding and removing shards. The job is not affected. Cluster instances: Configuration changes can cause transient connection disruptions. Realtime Compute for Apache Flink attempts to re-establish the connection. If these attempts fail, a failover is triggered.	Standard master-replica instances: No action required. Cluster instances: To prevent failovers and speed up adaptation, manually restart the job after configuration changes are complete.	No
ApsaraDB Tair (Tair Enterprise Edition)	Same as MySQL.	Same as MySQL.	No
ClickHouse	If `shardWrite` is set to `false`: the job is not restarted. If `shardWrite` is set to `true`: behavior depends on the `inferLocalTable` option.	Take action based on the value of `inferLocalTable`: `false` (default): Add the new node's IP address to the URL, then restart the job. `true`: Manually restart the job. The local table node is inferred automatically.	No
ApsaraMQ for RocketMQ	ApsaraMQ for RocketMQ 4.x: Realtime Compute for Apache Flink performs a failover so the job adapts to partition count changes. ApsaraMQ for RocketMQ 5.x: VVR 8.0.6 or earlier: Does not automatically adapt to partition count changes. A manual restart is required. VVR 8.0.7 or later: Automatically adapts to partition count changes.	Automatic adaptation may cause repeated data consumption. If this is unacceptable, cancel the job before changing the partition count, then restart it from the last checkpoint once changes are complete.	Yes
Tablestore	No data exists in the connector or is held in the buffer.	N/A	Yes
SelectDB connector	Data writing is not affected.	N/A	No
MongoDB	During data reading, topology changes cause the `134 - ReadConcernMajorityNotAvailableYet` error. This error is non-retryable.	Cancel the job before making changes that affect cluster topology. Restart the job after the changes are complete and the cluster has returned to normal.	Yes
Object Storage Service	Not applicable. An independent metadata layer describes data structures and status, and no scaling events affect these systems.	N/A	Yes
Iceberg connector	Not applicable.	N/A	Yes
Apache Paimon connector	Not applicable.	N/A	Yes
Hudi (retiring)	Not applicable.	N/A	Yes
Print	Not applicable. This connector is for testing only.	N/A	N/A
Blackhole	Not applicable. This connector is for testing only.	N/A	N/A
Datagen	Not applicable. This connector is for testing only.	N/A	N/A
Faker	Not applicable. This connector is for testing only.	N/A	N/A