jiangxt2 opened a new issue, #54921: URL: https://github.com/apache/arrow/issues/54921
### Describe the enhancement requested Add a public API to iterate over `RecordBatch` objects from selected stripes of an ORC file, with a configurable maximum number of rows per batch and column projection. [`ORCFile`](https://arrow.apache.org/docs/python/generated/pyarrow.orc.ORCFile.html) currently exposes whole-file and whole-stripe reads. When an application already knows which stripes it needs, it can call `read_stripe()` for each one, but each selected stripe is materialized as a single Arrow batch before processing can start. Slicing that result into smaller batches does not avoid the initial allocation. For large stripes, this creates memory pressure and unnecessary conversion when the consumer stops early. The Dataset API already supports streaming full-file ORC scans. The missing capability is combining explicit stripe selection with batch iteration through the public Python API. A possible interface, subject to API review, is `ORCFile.iter_batches(batch_size=65536, stripes=None, columns=None)`, analogous to `ParquetFile.iter_batches(row_groups=...)`. It should yield projected batches of at most `batch_size` rows without first materializing an entire selected stripe, and allow consumption to stop before the remaining rows are converted. Existing `read()` and `read_stripe()` behavior would remain unchanged. The [C++ ORC adapter](https://github.com/apache/arrow/blob/373d64c02cb23663b63c5853099fca43a46c39fc/cpp/src/arrow/adapters/orc/adapter.h#L132) already provides `Seek()`, `NextStripeReader(batch_size)`, and a full-file `GetRecordBatchReader()`, but these batch-reading entry points are not exposed by the ORC Python bindings. This request covers reading caller-supplied stripe IDs, independently of automatic statistics-based pruning or distributed task splitting. The batch size would limit output row counts; ORC dictionaries and decoder buffers could still require additional memory. Related work: [#49288](https://github.com/apache/arrow/issues/49288) covers Dataset stripe subsetting, and [#49360](https://github.com/apache/arrow/issues/49360) covers statistics and selective reads. This request focuses on Python batch iteration over selected stripes. Drafted with assistance from Codex. ### Component(s) Python, C++ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
