mzw111 commented on issue #10203:
URL: https://github.com/apache/seatunnel/issues/10203#issuecomment-5220985984
thanks for the guidance. I reviewed the current CDC flow and understand the
scope now.
- my plan is to keep the feature generic in the CDC base layer, while
implementing and testing it first with MySQL.
- On startup or checkpoint/savepoint recovery, newly matching table should
follow normal snapshot + incremental CDC flow. During normal job execution, new
matching table detected from MySQL binlog should be added for incremental CDC
only, without restarting the job.
- for keeping checkpoint and restore behavior safe, the enumerator will
remain responsible for admitting tables and tracking which incremental
split/reader owns each table. That ownership and the admitted table set will be
included in checkpoint state, so after failover we do not lose the table or
process it through multiple readers.
- I will keep this disabled by default behind an explicit configuration
option to preserve existing behavior. I will also add tests for matching/non
matching tables, checkpoint recovery, and duplicate admission.
please let me know if this direction matches your expectations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
