cshuo opened a new pull request, #20241: URL: https://github.com/apache/hudi/pull/20241
### Describe the issue this Pull Request addresses Flink's global record index creates a cache for each checkpoint, but cleanup previously ran only when estimated record memory exceeded the cache budget. Empty caches contribute zero estimated record bytes, so repeated idle checkpoints could accumulate cache objects indefinitely even after they became eligible for eviction. ### Summary and Changelog #### Commit: fix(flink): reclaim empty checkpoint caches in record index (`db3a81ece31`) - Scan eligible checkpoint caches from oldest to newest, closing and removing empty caches regardless of memory pressure. Continue scanning past non-empty caches so they do not block cleanup of newer empty caches. - Preserve the retention boundary and memory-pressure eviction of non-empty caches. Use `isEmpty()` to retain caches containing only spilled records. - Calculate total memory once per cleanup pass and decrement it as caches are removed. Keep the existing cleanup triggers: checkpoint cache creation and periodic record updates. - Add regression tests for repeated empty checkpoints, non-empty and inflight cache retention, cache closure, and disk-only entries. Validation: `TestRecordIndexCache` and `TestGlobalRecordLevelIndexBackend` passed all 24 tests with the Flink 1.20 profile and Java 11 (zero failures, errors, or skipped tests). ```bash mvn -nsu -pl hudi-flink-datasource/hudi-flink -am -Pflink1.20 -Dtest=TestRecordIndexCache,TestGlobalRecordLevelIndexBackend -Dsurefire.failIfNoSpecifiedTests=false test ``` ### Impact Prevents empty checkpoint cache accumulation during idle periods when the retention boundary advances, and reduces unnecessary cache traversal on lookups. No public API, configuration, or storage format changes. ### Risk Level Low. The change is limited to global record index cache eviction. Checkpoints at or above the retention boundary remain protected, and regression tests cover non-empty, inflight, and disk-only caches. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
