alamb commented on issue #15191: URL: https://github.com/apache/datafusion/issues/15191#issuecomment-5258446654
> We need to be careful about handling nullable sort columns. The min/max statistics don't include null values, and the result needs to respect the null-ordering behaviour. Any null values in a sort column of a partition are likely to invalidate the optimization, unless they only appear in the last partition when nulls are ordered last for example, or if they're only in a secondary sort column and the non-overlapping property can be proven based on sort columns with a higher priority. 👍 At least for timeseries usecases, the data for sort key is always non nullable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
