Hi, I use Apache Parquet occasionally, primarily from Python. I recently wrote a small library that uses custom fault handling to map address ranges lazily to derived content, and realized that this might also be useful for reading Parquet files.
The library can expose an entire column through an mmap-like interface, loading row groups lazily as their values are accessed. This provides an API that hides row-group boundaries from the caller while retaining their lazy-loading behavior. I have a working example for reading an integer-valued Parquet column as a lazy, zero-copy NumPy array. This makes it possible to write ordinary vectorized operations over the entire column without handling row groups explicitly. Does this seem useful for real-world Parquet workflows? Or is there usually intrinsic value in making row groups explicit in application code? The library and example are available here: https://github.com/mandolaerik/faultcache Thanks, Erik
