Hi All, There is an issue right now where Kafka IO's SDF implementation isn't resuming properly when the pipeline restarts https://github.com/apache/beam/issues/21730.
In addition, there was an issue where Kafka SDF wasn't committing properly when 'commit in finalize' was specified, and I believe there may also be an issue with restriction tracking, though I haven't confirmed that. Because of these issues, I don't have a good degree of trust in Kafka SDF, and because these aren't edge cases I don't believe there are many Kafka SDF users at the moment. As such, I've raised https://github.com/apache/beam/pull/22261 to disable Kafka SDF temporarily (Thanks @Chamikara Jayalath <chamik...@google.com> for setting up the new experiment that will allow Kafka Unbounded to continue working wrapped in SDF for runners that require SDF) In addition, https://github.com/apache/beam/issues/22303 tracks the work required to test, fix, and ensure that Kafka SDF is stable. Thanks, John
