villebro commented on issue #2843: URL: https://github.com/apache/datafusion-ballista/issues/2843#issuecomment-6883632937
@milenkovicm I really like this, and I feel auto would be a great default execution mode, as it's probably what fits the expectations and preferences typical users have. My quick thoughts: - I assume the decision would need to be made from the logical plan as opposed to the physical plan, as currently the physical plan depends on the mode. - Routing Copy and DML to staged execution makes sense. - The trait can't only take logical plans, as clients can also submit physical plans directly. So when a logical plan is available, use that, otherwise pass in the physical plan. - If auto becomes the default, it should fall back to staged on pull-staged schedulers, and we should document that pipelined jobs skip AQE. Beyond this I feel it would be a great idea to support multiple strategies through some form of registry, so we could add more strategies in the future (both in vanilla Ballista, but also on custom crates). That way one wouldn't have to settle for a single strategy, but could have many, and set one either at startup as the default, or per session or query through config. I'm planning on doing this for scheduler HA strategies, and I feel this type of extensibility would be valuable in all areas of Ballista. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
