DanielLeens commented on issue #13157: URL: https://github.com/apache/seatunnel/issues/13157#issuecomment-6687124960
Thanks for spelling out the desired data and failure semantics. This should be tracked as a schema-evolution design item, not started as an isolated SQL-transform implementation. There are already two active, non-final pieces of the same end-to-end contract: - #12284 translates schema changes through the SQL transform, but it is Zeta-oriented, remains open, and its current Build is failing. - #11912 coordinates schema changes across parallel Flink sink writers, but it remains open, has a current conflict, and its latest Build timed out. Neither is proof that a MySQL CDC -> Flink SQL -> JDBC pipeline has a complete, recoverable contract. The umbrella discussion #11402 explicitly requires agreement on per-table event/data ordering, checkpoint boundaries, sink application and replay before the capability is declared complete. Please start with a short STIP/design proposal linked to #11402 and a `[email protected]` discussion. It needs to make these decisions testable: 1. The exact first supported scope: source, SQL projection forms, JDBC sink, Flink versions, and the add/drop/rename/type-change operations that are accepted or rejected. 2. The ordering boundary: old-schema records must drain before the schema event is applied, and each parallel sink writer must observe one schema epoch before any post-change record is accepted. 3. Restore/replay behavior when an external JDBC DDL has succeeded but the corresponding checkpoint has not, including the idempotency mechanism for every supported mutation. 4. SQL output semantics: a projected-away upstream column must not create a sink column, while a projected type or alias must produce an event matching the transform output rather than the upstream table. 5. Recovery evidence: an E2E matrix with parallel sinks plus failures before apply, during apply, after apply/before checkpoint, and restore on both sides of the schema-change boundary. Once that contract is agreed, split the implementation so it can compose with #12284 and #11912 rather than duplicate either branch. No labels or assignments were changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
