jtuglu1 commented on issue #9780: URL: https://github.com/apache/druid/issues/9780#issuecomment-4278313486
> Hey [@jtuglu1](https://github.com/jtuglu1), I finally got a moment to push up the [3.x connector](https://github.com/JulianJaffe/druid/tree/spark_3_connector), with partitioning delegated to Spark's AQE so users don't have to. Awesome! Thank you! I'll start hacking on this. > Re the conversation you and Gian had above: > > > With that structure, integrating as a Unix tool rather than a Java library, we'd run in a different JVM, presumably a newer version. > > Spark really, _really_ does not do well with this approach since there's no good way to tell a Spark executor "reserve this amount of off-heap memory but not for Spark tasks". I think we might be able to come up with a solution here, provided everything else works. Whether that be providing the Spark extension as a build-only extension in the source tree (e.g. excluded from Druid release jars, requiring users to build themselves with a compatible JDK). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
