zephyr45 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6539785580
Thanks @SEZ9 and @DanielLeens — I addressed the blocking findings in `585f69b7`. - Removed the Zeta-specific modulo ownership rule. A restoring writer now adopts one non-legacy namespace from the old states actually assigned to it (preferring its exact index when present, otherwise deterministically choosing one). Since an old state/namespace is assigned to one restoring writer, siblings neither adopt nor clean the chosen current namespace. This covers Flink's contiguous 3-to-2 assignment as well as Zeta's modulo assignment without a cleanup/new-writer race. - Each restored state is parsed relative to its own saved base prefix, so scale-down works when `transaction_prefix` was omitted and the assigned states contain different generated prefixes. - Transaction IDs now use a contiguous per-writer sequence independent of checkpoint IDs. Every writer, including a stateless scale-up writer, scans and fences its current namespace from sequence 1 before opening work; there is no 1-to-10 gap hidden behind the first unused ID. - Spark 2.4 and 3.3 now use stable `partitionId` rather than retry-specific `taskId` as the sink-writer index. The Spark unit regression passes `partitionId=2`, `taskId=99`, and verifies index 2. - The English and Chinese option rows now link to and state the `<prefix>-<writer>-<sequence>` shape. I deliberately did not invent a global owner for legacy shared-prefix cleanup: mixed historical prefixes provide no engine-neutral cross-writer coordinator. That low-severity path can do redundant fencing, but correctness is preserved. Current-head validation: - Kafka connector unit suite: 143 passed; `KafkaSinkWriterTest` 11/11. - Spark 3.3 `SparkSinkTest`: 2/2. - Affected Kafka/Spark package reactor: 30/30 modules. - Documentation checks: no failures/errors. - Real Kafka E2E: Zeta parallel exactly-once + 3-to-2 restore, 2/2. - Real Kafka E2E: parallel exactly-once on Spark 2.4.6 and Spark 3.3.0, 2/2. - Spotless and `git diff --check` passed. GitHub had disabled inherited workflows on the fork despite the API reporting Actions enabled. I re-enabled them and queued the exact-head Build: https://github.com/zephyr45/seatunnel/actions/runs/41174376848 (head `585f69b76ed066ec7c270f688fbb9d7e9753adc5`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
