timsaucer commented on PR #110: URL: https://github.com/apache/datafusion-java/pull/110#issuecomment-4719876697
> In this approach we are basically building a mini ADBC but it's a constrained subset of the things ADBC is trying to do Yes, but I'm starting to think more about this. I did some analysis with my agent of where the gaps in the ADBC were and it was suggesting the problems with not having explicit push down filters. However I think if we use the substrait interface instead of the SQL interface then that argument disappears. I'm slightly outside my comfort zone on some of these aspects. What I'm going to try next - instead lean in towards your suggestion of ADBC and see if that can support all of the spark options I'd need and still get the pushdown. Also I need to make sure the task/partition mapping would still work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
