Dandandan commented on PR #2336: URL: https://github.com/apache/datafusion-ballista/pull/2336#issuecomment-5327473461
> Would it be possible to compare ballista and spark plans? it might give us some clue where to look, is it planning or physical operators or infrastructure or all previous the problem I looked at SF=10 plans yesterday, many plans looks pretty good but there are some categories: * Wrong join order (most of them before AQE) * No dynamic filters (shared) * Not always broadcast joins when beneficial (could be join order as well, but also we don't support broadcasting all joins yet) * Also I suspect at higher `target_partitions` we create too many shuffle files which might lead to connection issues / timeouts etc. (would need to replicate this) in e.g. SortPreservingMergeExec * Semi-join pushdown (Q18 could benefit) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
