MashrafeeAryan opened a new issue, #55287: URL: https://github.com/apache/arrow/issues/55287
### Describe the enhancement requested ### Describe the enhancement requested When writing a `FixedSizeListArray` (for example embeddings, coordinates or RGB tuples), Parquet C++ uses the same generic path as variable-length lists. `MultipathLevelBuilder` (`cpp/src/parquet/arrow/path_internal.cc`) walks the `FixedSizeListNode` / `FixedSizedRangeSelector` path node by node. It materializes one `int16_t` definition level and one repetition level per leaf value. Then `LevelEncoder::Encode` (`column_writer.cc`) RLE/bit-pack encodes those buffers again one value at a time. For a list of size `n` with no nulls, both level streams are fully predictable: - def levels: a constant `max_def_level`, which is a single RLE run - rep levels: the periodic pattern `0, 1, …, 1` (`n - 1` ones), repeated for every record So we spend time and memory building and encoding 2 × `num_values` levels that could be written directly. This is the write-side counterpart of the read-side fast path described in [Fast path for fixed-length lists in Parquet](https://www.morling.dev/blog/fast-path-for-fixed-length-lists-in-parquet/). ### Proposal Add a writer fast path that applies when: - the leaf column sits under a `FixedSizeListArray` (optionally also a `ListArray` / `LargeListArray` whose offsets show every list has the same length), and - the list has no null slots, the child values have no nulls, and there are no nullable or repeated ancestors (at least for the first version). In that case: 1. Skip the generic path traversal and produce the def/rep levels directly. Ideally they would go straight into the RLE encoder as runs (one def-level run per page; a precomputed periodic rep-level pattern per page) without materializing `int16_t` buffers. 2. Pass the child values slice straight to `WriteArrowDense` (it is already contiguous: `values()->Slice(value_offset(0), length * n)`). 3. Compute the row count, page and record boundaries, and `SizeStatistics` level histograms arithmetically (`num_rows = num_values / n`) instead of scanning rep levels. Output must be **byte-identical** to the existing path, so files stay fully readable by every existing reader. No format or spec change is involved. ### Acceptance criteria - Round-trip tests for `fixed_size_list<T, n>` covering several `n` (1, 3, 8, 9, 16, 768), several physical types, V1 and V2 data pages, dictionary on/off, and a range of page and row-group sizes. - A test that the fast path and the generic path produce identical column chunks, including levels, page boundaries and statistics. - The generic path still handles null lists, null elements and nested cases. - A benchmark in `reader_writer_benchmark.cc` comparing writing `fixed_size_list<float, n>` against writing the same values as a flat column, before and after the change. ### Context - Related read-side issue: #61 - Existing report of slow fixed-size-list IO: `apache/arrow#34510` - Parquet dev list discussion on a `FIXED_SIZE_LIST` logical type: https://lists.apache.org/thread/qhq33wg1loxhymsyqjnsfsrd82qnv43m. This issue works independently of that discussion, because it needs no spec change. If a spec-level fixed-size marker lands later, the writer can emit it from this same path. ### Component(s) C++, Parquet ### Component(s) C++, Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
