SEZ9 commented on issue #9063: URL: https://github.com/apache/seatunnel/issues/9063#issuecomment-6206764758
@goutamadwant thanks a lot for reproducing this and for picking it up — a clean repro on Spark 3.3 that fails once the first snapshot split finishes matches exactly the gap described here: the CDC reader needs to send `SourceEvent`s back to the enumerator, but the Spark translation path still routes the source through the plain parallel reader context, which throws the `UnsupportedOperationException`. A few asks so we can review the PR you linked efficiently: 1. Please describe in the PR how the source is now exposed through the coordinated path on Spark (how `SupportCoordinate` / `SourceEvent` are detected and handled in the translation layer), so reviewers can check it covers CDC sources generally and not only MySQL-CDC. 2. You mentioned parallelism > 1 is covered for both Spark 2.4 and 3.3 — please add (or point to) tests/e2e cases that exercise the snapshot-split completion and the snapshot-to-incremental phase transition with parallelism > 1, since that is where the event exchange actually happens. 3. Scoping out `readStream` micro-batch CDC with a clear error sounds reasonable for this fix; please state that limitation explicitly in the PR description and in the error message text so users on Spark understand what is and isn't supported. 4. If possible, include the job config you used for the local repro in the PR, so we can verify the fix against the original Mysql-CDC -> Hudi scenario on YARN cluster mode as well. @eyys you were assigned this earlier — if you have any partial work or findings, please share them on the PR thread so efforts can be combined; otherwise I'll move the assignment to @goutamadwant. Thanks again, this is a very welcome contribution. <!-- streview-comment:1786 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
