cshuo opened a new issue, #20116:
URL: https://github.com/apache/hudi/issues/20116
### Task Description
**What needs to be done:**
Avoid buffering record positions in `HoodieNativeLogFormatWriter` when
position metadata is not available or enabled for the append. Gate collection
in both `appendRecord()` and `appendDeleteRecord()` on
`baseFileInstantTimeOfPositions.isPresent()`.
**Why this task is needed:**
The native log writer streams records into the underlying file writers, but
currently maintains two additional `ArrayList<Long>` buffers:
- `appendRecord()` unconditionally adds `record.getCurrentPosition()` to
`dataRecordPositions`.
- `appendDeleteRecord()` adds an entry to `deleteRecordPositions` for every
deletion, using `-1L` when there is no base-file instant for positions.
- At flush time, `addRecordPositionsIfRequired()` returns without using
either buffer when the header lacks
`BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS`.
Consequently, appends without position metadata retain O(N) unused list
entries, where N is the total number of data and delete records accumulated
before a flush. Clearing the lists after a flush releases their elements but
retains the backing array capacity. Even when boxed values such as `-1L` are
cached, the list reference arrays still grow with the record count.
This is an avoidable heap-memory cost; no data correctness failure has been
observed.
**Reproduction:**
1. Construct a native log writer with `baseFileInstantTimeOfPositions =
Option.empty()`, as can occur when writing record positions is disabled or no
base file is available.
2. Append data records and/or delete records.
3. Inspect the corresponding position lists before flushing: they contain
one entry per appended record even though the footer will not use them.
**Expected behavior and validation:**
- With no base-file instant for positions, both position buffers remain
empty throughout the append and the footer contains no position metadata.
- With position metadata enabled, preserve the existing encoding behavior,
including data-record order, duplicate positions, and invalid-position handling.
- Regression tests should inspect the buffers before flush, in addition to
checking the resulting footer. Footer-only assertions do not detect this
unnecessary allocation.
A local fix with this coverage passed all 13 tests in
`TestHoodieNativeLogFormatWriter` using:
```bash
mvn -pl hudi-client/hudi-client-common -am \
-Dtest=TestHoodieNativeLogFormatWriter \
-Dsurefire.failIfNoSpecifiedTests=false \
-DskipITs -DskipSparkTests -DskipScalaTests test
```
### Task Type
Performance optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
