adriangb commented on issue #8078: URL: https://github.com/apache/datafusion/issues/8078#issuecomment-3392247494
> > TLDR is it's expensive to randomly sample compressed columnar storage like Parquet, but if you store a pre-sampled portion of the file e.g. as the last row group you can get very good estimates for all kinds of things (filter selectivity, cardinality of any column, etc.) > > Sounds like a good usecase for https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/ 🤔 Yep could be that! I was thinking maybe the last row group would be beneficial because (assuming the data is basically Parquet data) it avoids having to re-encode all of the metadata. Also sadly our Parquet reader cannot be pointed at a byte range of a file (I think that'd be easy to fix in a PR). But it does make it incompatible with other readers (if they don't know to skip the last row group...). Anyway that should be discussed in another thread I just wanted to share the idea as a possibly use case for more advanced statistics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
