rkrishn7 opened a new pull request, #27501:
URL: https://github.com/apache/datafusion/pull/27501

   ## Which issue does this PR close?
   
   - Related to #27498. Found during review of #27499.
   
   ## Rationale for this change
   
   Two `GroupValues::intern` paths can panic:
   
   - `GroupValuesColumn::emit(EmitTo::All)` replaces the column builders but 
keeps the hash map and collision lists. Interning a key seen before the emit 
finds a stale entry and indexes the empty builders: `index out of bounds: the 
len is 1 but the index is 2`. In-tree callers call `clear_shrink` after 
`emit(EmitTo::All)` or stop, so only code that reuses a `GroupValues` after a 
full emit hits this.
   - `ArrowBytesViewMap::insert_if_new` compares a long input with a stored 
entry that has the same hash and 4-byte prefix without checking the length. If 
the stored entry is inline, its bytes are read as a buffer index and offset: 
`range start index 1499027801 out of range for slice of length 0`. The values 
can't compare equal, so this never returns a wrong group, but it needs a 64-bit 
hash collision to happen and then panics.
   
   ## What changes are included in this PR?
   
   - `GroupValuesColumn::emit(EmitTo::All)` clears the hash map and collision 
lists along with the column builders.
   - `ArrowBytesViewMap::insert_if_new` skips entries whose length differs from 
the input, as `ArrowBytesMap` already does.
   
   ## What is the testing strategy for this PR?
   
   Two unit tests that fail on `main` and pass with this PR:
   
   - `intern_after_emit_all_starts_new_groups` interns, emits everything, 
interns again, and checks the new group ids in both streaming and vectorized 
modes.
   - `test_long_value_colliding_with_inline_value` interns an inline value and 
then a long value with the same prefix. Hashes only collide with the 
`force_hash_collisions` feature, which the extended CI job enables.
   
   ## Are there any user-facing changes?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to