ad1happy2go opened a new pull request, #20273: URL: https://github.com/apache/hudi/pull/20273
### Describe the issue this Pull Request addresses An incremental query with `hoodie.datasource.read.incr.fallback.fulltablescan.enable=true` whose start instant is archived silently returns **no rows**, or only some partitions' rows, when a completed non-write instant (rollback or clean) older than every active commit is still on the active timeline. Two components decide whether the start is archived, and they look at different timelines: - **The relation** (`MergeOnReadIncrementalRelationV1.startInstantArchived`, and `V2` via `queryContext.getArchivedInstants`) checks the **commit timeline**. It sees the start as archived and falls back to a full table scan. - **The file index** (`BaseHoodieTableFileIndex.isBeforeTimelineStarts`) checks the **whole active timeline**. The older rollback makes it treat the start as active, so it takes the incremental branch of `listPartitionPaths`. That branch lists only the partitions written by `findInstantsInRange()` on the write timeline. The log shows `listing full table (0 partitions)` and the scan returns nothing, or only the partitions touched by active commits in the range. This state is ordinary: the table version 6 archiver only archives rollback instants once there are more than `hoodie.keep.max.commits` of them, so a single rollback from a failed write outlives the commits around it. It also stays on the active timeline when the table is upgraded to version 9, so the V2 (completion-time) path is affected as well. Reproduced with a table written by 0.14.1 and read by 1.1.1 and 1.2.1, with a full table scan as fallback: | Window | Expected | Returned | |---|---|---| | Archived → archived | 500 | **0** | | Archived → latest | 1500 | **600** (2 of 5 partitions) | | Active → latest | 100 | 100 | The same table without the rollback returns the expected rows. ### Summary and Changelog The file index now checks whether the start is archived against the **completed write timeline**. That is the timeline its incremental partition listing reads, so a stale rollback or clean no longer makes an archived start look active. - `BaseHoodieTableFileIndex.isBeforeTimelineStarts()` uses `getActiveTimeline().getWriteTimeline().filterCompletedInstants()`. This applies to both the requested-time path and the completion-time path. - Unit test `BaseHoodieTableFileIndexTest#testIncrementalStartBeforeFirstWriteIsArchivedDespiteOlderNonWriteInstant` covers both query modes. It fails without the fix. - Functional test `TestIncrementalReadWithFullTableScan#testFallbackFullTableScanWithStaleRollbackOnActiveTimeline` writes a failed write, its rollback and archived commits rotating over 5 partitions. It then compares fallback incremental reads over archived, mixed and active windows with the expected commits and partitions. It covers COW v6, COW v6→v9 upgraded, and MOR v6. Without the fix all three fail; post-fix CI results are still pending, see the checklist. ### Impact Fallback incremental reads (V1 and V2, COW and MOR) return correct results on tables that carry such instants. Without the fallback, the relation already filters the listed partitions down to the ones changed in range, so results don't change. The file index may list more partitions in that case, from the metadata table. ### Risk Level low. This is a one-method change to the guard in front of the incremental partition listing. It can only make the file index take the existing "list all partitions" path more often, which is a superset of the incremental listing. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable - [ ] Post-fix run of the functional test and the existing incremental/CDC suites (in progress) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
