alamb commented on PR #20292: URL: https://github.com/apache/datafusion/pull/20292#issuecomment-4037988529
> @alamb do you see any way we can have NDV/distinct_count when generating test files via that tool? > > There has been a lot of activity and interest lately around NDV (including myself), it'd be nice to be able to track progress via this new test suite. @asolimando I think we would need to add support to arrow-rs for writing the distinct statistics first - https://github.com/apache/arrow-rs/issues/8608 One challenge we have had so far is that distinct count is often quite memory and CPU expensive to calculate (there are less intensive estimates too like hyper log log but the actual parquet statistics call for NDV). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
