zhaoyudi-creator opened a new pull request, #19951: URL: https://github.com/apache/hudi/pull/19951
### Describe the issue this Pull Request addresses With Flink streaming writes and the simple Bucket Index (non-NBCC), if a failover/restore happens while the *first* write to a partition is still uncommitted, the restarted task reads only the committed file-system view (still empty at that point) and mints a brand-new fileId for the bucket. This collides with the inflight fileId the coordinator later recommits, and the next bootstrap fails with `Duplicate fileId ... found`. See the [issue](https://github.com/apache/hudi/issues/19907) for the full reproduction timeline. ### Summary and Changelog Fixes a duplicate-fileId failure that can permanently break bootstrap of a simple Bucket Index partition after a failover. The fix has `bootstrapIndexIfNeed()`, in addition to reading the committed view, query the coordinator for the fileIds it still holds pending (checkpointed but not yet committed) and reuse them, so a restarted task adopts the original fileId instead of minting a new one. I'd appreciate a maintainer's read on the direction before I invest further. An alternative would be a deterministic fileId (like NBCC's `-0000-...`), which is a smaller change but changes the on-disk fileId format, diverges from the non-NBCC behavior of Spark/bulk-insert, and cannot recover an already-existing inflight bucket that used a random id across an upgrade. The coordinator-query approach here keeps the format unchanged and naturally covers rescale (after a bucket's owner changes, the new owner adopts the pending fileId). If this direction is acceptable, I'll extend the test coverage (see below). Planned follow-up tests if the approach is accepted: 1. Mixed partition — some buckets committed and others only pending within the same partition (committed view wins, overlay does not overwrite an existing fileId). 2. Cross-partition isolation — multiple partitions each holding pending fileIds where bucket numbers repeat across partitions (the query filters by partition and does not cross fileIds). 3. Multi-task ownership filter / rescale — parallelism > 1 and parallelism-change scenarios (likely better suited as an IT). ### Impact No public API or user-facing change. Adds one internal coordination request that is read-only and issued only once per partition on first bootstrap (guarded by `bucketIndex.containsKey(partition)`). It is off the per-record hot path and adds no blocking to the commit pipeline, so it does not affect the async instant-generation optimization. NBCC and other index types are unaffected. ### Risk Level <!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk. If medium or high, explain what verification was done to mitigate the risks. --> ### Documentation Update <!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none". - The config description must be updated if new configs are added or the default value of the configs are changed. - Any new feature or user-facing change requires updating the Hudi website. Please follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. --> ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
