milenkovicm commented on PR #1461: URL: https://github.com/apache/datafusion-ballista/pull/1461#issuecomment-3921569620
> Results look great! Good call on the standalone. > > Separately, curious if either of you have played with tuning the loop interval for pull-based? I see this condition currently: > > https://github.com/apache/datafusion-ballista/blob/0396c58aa58a546395fa59bf6658e17c621c0cc3/ballista/executor/src/execution_loop.rs#L206 > > Looks like 100ms (without an active job) is the norm. Would it be worth making this configurable? yes, if changed to 10ms perfromance get better a lot, but still as fast as other. I guess we could make some kind of back-off strategy here, if there are tasks sleep time goes down. also there is another one https://github.com/milenkovicm/datafusion-ballista/blob/0396c58aa58a546395fa59bf6658e17c621c0cc3/ballista/core/src/execution_plans/distributed_query.rs#L369-L370 perhaps we could make this push based rather than pull based. I'll decrease that sleep to 50, but we could have similar back off strategy here as well wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
