bharath-techie commented on issue #24288:
URL: https://github.com/apache/datafusion/issues/24288#issuecomment-5269340232
@alamb @etseidl, one challenge I faced was assembling `ParquetMetaData` from
granularly cached page-index entries.
The metadata cache continues to serve the file and row-group metadata
through a shared `Arc<ParquetMetaData>`. The scoped caches separately store
`column-index` and `offset-index` entries. For each query, I need to combine
the shared footer metadata with only the page indexes required by that query.
There were two difficulties:
1. Attaching page indexes requires an owned `ParquetMetaData`. Because the
footer remains shared by the metadata cache, `Arc::try_unwrap` fails and the
fallback is to clone the complete metadata. This copies
every `RowGroupMetaData` and `ColumnChunkMetaData` just to replace the
page-index fields.
2. `ParquetColumnIndex` and `ParquetOffsetIndex` were dense
`[row_group][column]` matrices. A scoped index therefore required placeholders
for unrequested entries. Column indexes had `ColumnIndexMetaData::NONE`, but
offset indexes had no equivalent, so I had to construct conservative synthetic
offset-index entries.
#10653 addresses the second problem by changing both matrices to contain
`Option<T>`. This should remove the need for synthetic placeholders and allow
missing entries to fall back conservatively.
The first problem remains. Even with optional page-index cells, combining
a shared cached footer with query-specific indexes still requires cloning
`ParquetMetaData` before calling `into_builder()`.
I prototyped storing `row_groups` behind an `Arc`, which makes this clone
share the expensive row-group and column-chunk metadata. The public API remains
unchanged, and attaching indexes becomes inexpensive because only the outer
metadata and page-index fields are rebuilt.
Would this be a reasonable follow-up to #10653, or would you prefer page
indexes to be passed separately from the shared footer metadata so that
rebuilding `ParquetMetaData` is unnecessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
