zhuqi-lucas commented on PR #18112:
URL: https://github.com/apache/datafusion/pull/18112#issuecomment-3418422210
It seems always request 2 times to s3 at lease if we enable page index, i
will investigate more:
```rust
/// Attempts to (asynchronously) parse the footer metadata (and
optionally page indexes)
/// given a [`MetadataFetch`].
///
/// See [`Self::with_prefetch_hint`] for a discussion of how to reduce
the number of fetches
/// performed by this function.
#[cfg(all(feature = "async", feature = "arrow"))]
pub async fn try_load<F: MetadataFetch>(&mut self, mut fetch: F,
file_size: u64) -> Result<()> {
let (metadata, remainder) = self.load_metadata(&mut fetch,
file_size).await?;
self.metadata = Some(metadata);
// we can return if page indexes aren't requested
if self.column_index == PageIndexPolicy::Skip && self.offset_index
== PageIndexPolicy::Skip
{
return Ok(());
}
self.load_page_index_with_remainder(fetch, remainder).await
}
```
I am not sure if with_prefetch_hint can make the eager loading page index
parquet metadata to happen in one request...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
