wuhainan opened a new issue, #13157:
URL: https://github.com/apache/seatunnel/issues/13157

   ### Search before asking
   
   - [x] I had searched in the 
[feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22)
 and found no similar feature requirement.
   
   
   ### Description
   
   ### Background
   
   SeaTunnel supports schema evolution for CDC pipelines. However, when a 
pipeline runs on the Flink engine and includes a SQL Transform, schema change 
events are not currently handled by the transform as schema changes.
   
   This makes it difficult to use data transformation and automatic schema 
evolution together in a Flink-based CDC pipeline.
   
   ### Feature request
   
   Support schema evolution through SQL Transform when running SeaTunnel jobs 
on the Flink engine.
   
   When a source emits a schema change event, the pipeline should:
   
   1. Deliver the schema change to the transform and update its runtime schema.
   2. Propagate the resulting output schema change to the sink when needed.
   3. Keep transformed records consistent with the sink schema before and after 
the change.
   4. Fail with a clear error when a schema change cannot be safely handled.
   
   The behavior should also be verified with parallel execution and 
checkpoint/savepoint recovery.
   
   This request focuses on the Flink execution path and can build on the SQL 
Transform schema handling in #12284.
   
   ### Usage Scenario
   
   A MySQL CDC job runs on the Flink engine and synchronizes data to TiDB 
through the JDBC sink.
   
   The job uses a SQL Transform to mask sensitive fields, such as phone 
numbers, before writing them to TiDB. When the source table schema changes, we 
want the pipeline to handle supported schema changes without manually stopping 
and rebuilding the job.
   
   For example, if a source column used by the transform changes type, the 
pipeline should update the transform and sink schemas as appropriate. If a 
change cannot be applied safely, the job should fail with a clear error.
   
   The behavior for newly added columns should follow the SQL projection: 
columns that are not part of the transform output should not be written to the 
sink automatically.
   
   ### Related issues
   
   - #8253 — SQL Transform support for schema changes.
   - #12284 — SQL Transform schema change handling for the Zeta engine.
   - #11752 — Schema change propagation to parallel sink writers on Flink.
   - #11402 — Discussion of schema change ordering and recovery.
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to