adamreeve commented on issue #15191: URL: https://github.com/apache/datafusion/issues/15191#issuecomment-5248918478
A couple of things I've found while experimenting with this so far: 1. We need to be careful about handling nullable sort columns. The min/max statistics don't include null values, and the result needs to respect the null-ordering behaviour. Any null values in a sort column of a partition are likely to invalidate the optimization, unless they only appear in the last partition when nulls are ordered last for example, or if they're only in a secondary sort column and the non-overlapping property can be proven based on sort columns with a higher priority. 2. When there are multiple sort columns, validating the non-overlapping property using the min/max statistics might reject partitions and consider them overlapping when they are not (a false negative). For example, say we have files ordered by `day` and `hour`. `0.parquet` has `(day, hour)` values covering the range `[(1, 9), (2, 8)]`, and `1.parquet` covers `[(2, 9), (3, 8)]`. `0.parquet` might have min statistics of `(1, 0)` and max statistics of `(2, 23)`, and `1.parquet` have min statistics of `(2, 0)` and max statistics `(3, 23)`. Based on the min/max statistics alone, we have to assume that these files might overlap and so we can't apply the optimization. I think this is probably acceptable as it's somewhat of an edge case. And maybe we can use the technique suggested in https://github.com/apache/datafusion/issues/15191#issuecomment-2722432514 to still form some ProgressiveEval streams and reduce the amount of merging required. Ideally we could have access to the sort-column values from the first and last rows though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
