When you modify the ORDER BY clause or the SELECT field list in a Flink SQL deployment, the change may or may not break compatibility with existing state data. Whether compatibility is preserved depends on two factors: the sort time attribute (proctime or rowtime) and the type of modification (SELECT fields vs. sort keys or direction).
Compatibility matrix
Use this table to determine whether a planned SQL change is safe before applying it to a running deployment.
| Sort time attribute | Change type | State compatibility |
|---|---|---|
proctime |
Add or remove a field in SELECT |
Compatible |
| Sort keys (non-time columns) | Add or remove a field in SELECT |
Compatible |
rowtime |
Add or remove a field in SELECT |
Incompatible |
| Any | Modify sort keys or sort direction in ORDER BY |
Incompatible |
"Compatible" means the deployment can resume from an existing checkpoint or savepoint without reprocessing historical data. "Incompatible" means the deployment cannot restore from the previous state and must be restarted from scratch.
Running example
All examples in this topic use the following base table and extend or modify it to illustrate each scenario.
-- Base table used throughout this topic
select a, b, c from MyTable order by <sort-expression> asc;Compatible changes
Change SELECT fields when sorting by proctime
Adding or removing a field from SELECT is safe when the sort expression uses proctime.
-- Original SQL statement:
select a, b, c from MyTable order by proctime asc;
-- Add field d to SELECT. The deployment remains fully compatible with state data.
select a, b, c, d from MyTable order by proctime asc;Change SELECT fields when sorting by sort keys
Adding or removing a field from SELECT is also safe when the sort expression uses non-time sort keys.
// Original SQL statement:
select a, b, ts from MyTable order by ts asc;
// Remove field ts from SELECT. The deployment remains fully compatible with state data.
select a, b from MyTable order by ts asc;Incompatible changes
The following changes break state compatibility regardless of the sort time attribute. After either of these modifications, the deployment cannot restore from the previous state.
Change SELECT fields when sorting by rowtime
Unlike proctime, modifying SELECT fields when sorting by rowtime makes the deployment incompatible with the state data.
-- Original SQL statement:
select a, b, c from MyTable order by ts asc;
-- Add field d to SELECT. The deployment becomes incompatible with state data.
select a, b, c, d from MyTable order by ts asc;Modify sort keys or sort direction in ORDER BY
Changing the sort keys or sort direction in ORDER BY always breaks state compatibility.
-- Original SQL statement:
select a, b, c from MyTable order by ts asc;
-- Add field a to ORDER BY. The deployment becomes incompatible with state data.
select a, b, c from MyTable order by ts asc, a desc;FAQ
Can I add a column to SELECT without breaking state?
It depends on the sort time attribute. If the deployment sorts by proctime or by non-time sort keys, adding or removing SELECT fields is safe. If it sorts by rowtime, the same change breaks state compatibility.