GitHub user wgtmac added a comment to the discussion: Thread Safety for parquet::arrow::FileReader::GetRecordBatch
I don't think we can do this blindly. Codex has spotted evidence like below: -------- With the current default settings, concurrent calls to GetRecordBatchReader() on the same FileReader are not safe. ArrowReaderProperties::pre_buffer defaults to true, and each call invokes ParquetFileReader::PreBuffer(). That method replaces SerializedFile::cached_source_ and clears/rebuilds prebuffered_column_chunks_, while GetRowGroup() reads both without synchronization. Different row-group IDs do not avoid the race because this state is file-wide. If pre-buffering is disabled before constructing the FileReader, then separate RecordBatchReader instances for different row groups appear safe to consume concurrently in the current implementation, provided that: - the FileReader outlives all returned readers; - its properties are not modified concurrently; - each RecordBatchReader is consumed by only one thread; and - the underlying RandomAccessFile follows the thread-safe ReadAt() contract. GitHub link: https://github.com/apache/arrow/discussions/51369#discussioncomment-18477425 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
