adriangb commented on PR #12606: URL: https://github.com/apache/datafusion/pull/12606#issuecomment-2374868022
> It might also be good to look at https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html#method.literal_guarantees which you might be able to use to apply you index I admit I'm still a bit confused about `LiteralGuarantees` but it seems to me that to use them I'd have to have all of the data in memory. The whole point of this is that I can store a relatively large amount of data (say 1M 16 character strings) and rip through filtering them by letting the system storing them (in my case a Postgres database with a GIN index on the array column) give me back just the row groups that matched the predicate, without ever moving all of that data over the wire to build `LiteralGuarantee`s from it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
