zephyr45 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6544414185
Thanks @SEZ9 and @DanielLeens — I found and fixed two residual correctness gaps in the previous update with `d8143b58`. - A fresh writer namespace no longer reuses the bare current subtask index. `KafkaSink` now creates a generation token once and serializes it with the sink graph. Task retries therefore reuse and fence the same namespace, while a later rescale/execution generation gives a stateless writer a namespace distinct from any old namespace adopted from restored state. This covers the repeated Flink 3 -> 2 -> 3 case where old namespace 2 can be adopted by writer 1 before writer 2 becomes stateless again. - When `transaction_prefix` is omitted, the generated job prefix is also chosen once in `KafkaSink` and serialized with the graph instead of being regenerated independently by each writer construction. A stateless retry can now identify and fence its previous attempt. Restore remains compatible with historical states whose writers saved different generated prefixes. - The parser accepts both the previous numeric writer namespaces and the new opaque generation-scoped namespaces; `KafkaSinkState` and its serialized fields are unchanged. - EN/ZH connector and incompatible-change docs now state that `<writer>` is opaque, stable across task retries, and not necessarily the current subtask index after rescaling. - Added regressions for repeated scale-down/scale-up namespace isolation, generated-prefix stateless retry through sink serialization, and legacy no-prefix migration. Final local validation on Java 11: - Kafka connector unit suite: 146 passed; `KafkaSinkWriterTest` 14/14. - 16-module affected dependency reactor: package succeeded. - Documentation checks: 7 run, 0 failures/errors (3 skipped). - Repository-wide Spotless and full `-DskipTests verify`: passed. - `git diff --check`: passed. The Docker daemon is unavailable in my local environment, so I did not claim a new local real-broker E2E result for this follow-up. The exact-head fork Build is running at https://github.com/zephyr45/seatunnel/actions/runs/41208442957 (head `d8143b588329f87ef2a123936165fba87a2d22b8`). The previously noted duplicate legacy shared-prefix probing remains an engine-neutral efficiency limitation, not a correctness gap: historical shared states contain no globally unique cleanup owner, and selecting writer 0 would miss distinct historical generated prefixes assigned elsewhere. Cleanup is still deduplicated within each writer and remains safe across writers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
