Minghan2005 opened a new pull request, #24004:
URL: https://github.com/apache/datafusion/pull/24004
## Which issue does this PR close?
- Closes #23978.
## Rationale for this change
The Parquet opener intentionally loads footer-only metadata first so
row-group
statistics can decide whether page-index I/O is necessary. When the page
index
is needed later, however, the deferred byte-range load bypasses
`DFParquetMetadata::fetch_metadata`, so the augmented metadata was not
written
back to `FileMetadataCache`. Reopening the same file therefore repeated the
page-index request.
## What changes are included in this PR?
- Add a default no-op `ParquetFileReaderFactory::cache_metadata` hook so
factories with a metadata cache can retain metadata augmented after the
initial request without requiring changes from existing implementations.
- Have `CachedParquetFileReaderFactory` replace the existing footer-only
cache
entry after a deferred page-index load. As with the existing metadata path,
decrypted metadata is not cached.
- Call the hook after the opener finishes loading the page index.
- Add a regression test that verifies the cache entry gains the page index
and
that a second open performs zero reader byte-range I/O.
## Are these changes tested?
Yes.
- `cargo test -p datafusion-datasource-parquet` (159 passed)
- `cargo test -p datafusion-datasource-parquet page_index -- --nocapture`
- `cargo test -p datafusion-datasource-parquet --all-features
test_deferred_page_index_is_cached -- --nocapture`
- `cargo clippy --all-targets --workspace --features
avro,integration-tests,extended_tests -- -D warnings`
- `cargo clippy -p datafusion-datasource-parquet --all-targets
--all-features -- -D warnings`
- Full repository formatting, TOML formatting, license, spelling, Prettier,
workflow-install, and rustdoc checks
- `cargo-semver-checks` against `upstream/main` (223 checks passed)
The regression test measures the I/O invariant directly rather than relying
on
wall-clock timing, so no separate benchmark was added.
## Are there any user-facing changes?
`ParquetFileReaderFactory` gains a default no-op cache-update hook. Existing
implementations remain source-compatible, and the SemVer check reports that
no
version update is required. The default cached reader now avoids repeated
page-index requests when opening the same file.
## AI assistance
I used OpenAI Codex while implementing this change. I reviewed the code and
tests and understand the metadata-loading and cache-update path end to end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
