Skip to content

[FLINK-39198] Fix data loss during restoration from checkpoint in TIMESTAMP startup mode.#4297

Open
danzhewuju wants to merge 2 commits intoapache:masterfrom
danzhewuju:fix-loss-data-restore-from-checkpoint
Open

[FLINK-39198] Fix data loss during restoration from checkpoint in TIMESTAMP startup mode.#4297
danzhewuju wants to merge 2 commits intoapache:masterfrom
danzhewuju:fix-loss-data-restore-from-checkpoint

Conversation

@danzhewuju
Copy link

@danzhewuju danzhewuju commented Mar 3, 2026

Fix: https://issues.apache.org/jira/browse/FLINK-39198

What is the purpose of the change

Flink CDC creates an eventFilter inside the MySqlBinlogSplitReadTask. This filter is produced through the following method. In timestamp startOptions mode, the eventFilter will exclude events whose timestamps are earlier than the user-defined timestamp. However, if the job resumes from a savepoint or checkpoint and the system automatically sets the timestamp parameter, this approach can result in data loss.

In this scenario, the user configures a startOption with timestamp mode (which may be the default behavior, tied to the Flink job’s submission time) and restarts the Flink CDC job from a savepoint. If the savepoint contains MySQL binlog information including a ts_sec value, and that ts_sec is earlier than the user-defined timestamp, data loss may occur.

Brief change log

Prevent filtering out binlog events when resuming from savepoints/checkpoints with older ts_sec

Verifying this change

Test result:
Adding split(s) to reader: [MySqlBinlogSplit{splitId='binlog-split', offset={transaction_id=null, ts_sec=1772531889, file=.577482, pos=883008898, kind=SPECIFIC, gtids=, row=1, event=3, server_id=5212, timestamp_strictly=false}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, kind=NON_STOPPING, row=0, event=0, timestamp_strictly=false}, isSuspended=false}]
Creating an event filter that drops row mutation events occurring before the destination timestamp in seconds 1772531889

…ESTAMP startup mode.

Signed-off-by: danzhewuju <1668926294@qq.com>
Signed-off-by: danzhewuju <1668926294@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant