comphead opened a new issue, #26687: URL: https://github.com/apache/datafusion/issues/26687
### Describe the bug DataFusion is meant to skip validating spill files when it reads them back, because it wrote them itself (#15454). Since #21882 (55.0.0), `SpillReaderStream` reads them with `StreamDecoder::with_skip_validation(true)` ([mod.rs#L337-L339](https://github.com/apache/datafusion/blob/c6869995c11c0e6661106f9d188dede8672ecc69/datafusion/physical-plan/src/spill/mod.rs#L337-L339)). But arrow-ipc's `StreamDecoder` only passes that flag to dictionary batches ([stream.rs#L247-L277](https://github.com/apache/arrow-rs/blob/df4dc3ede59bd43b5f71847231212381d068bb72/arrow-ipc/src/reader/stream.rs#L247-L277)). Every spilled record batch is validated again, including a UTF-8 check of every string. ### To Reproduce A spilled `Utf8` value that is not UTF-8 fails to read back: ```rust let binary = BinaryArray::from_iter_values([b"\xa0\xa1".as_slice(), b"valid"]); let strings = unsafe { StringArray::new_unchecked(binary.offsets().clone(), binary.values().clone(), None) }; let batch = RecordBatch::try_from_iter([("s", Arc::new(strings) as ArrayRef)])?; let spill_file = spill_manager .spill_record_batch_and_finish(&[batch], "Test")? .unwrap(); collect(spill_manager.read_spill_as_stream(spill_file, None)?).await?; // Invalid argument error: Invalid UTF8 sequence at string index 0 (0..2): invalid utf-8 sequence of 1 bytes from index 0 ``` ### Expected behavior Spilled batches read back without validation, as before 55.0.0. In the `spill_io` benchmark, reading 100 spilled batches of 8192 rows with a `Utf8` column takes 9.4 ms on main and 8.2 ms without validation. ### Additional context arrow-ipc 56.1 through 60.0 and arrow-rs main have the same `StreamDecoder` code. arrow-ipc tests `skip_validation` with `StreamReader`, `FileReader` and `FileDecoder`, but not with `StreamDecoder`. Found while working on #26627. ### Willingness to contribute I plan to contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
