wForget commented on PR #7539:
URL: https://github.com/apache/kyuubi/pull/7539#issuecomment-4893476399
> (!hasExpandingOperators && maxStageSize < sessionAdvisorySize *
tolerableSmallFileNum)
If I understand correctly, this is because CoalescePartitions will cause the
downstream shuffle to always approach sessionAdvisorySize, so we can remove
this rebalance. (should we check the
`spark.sql.adaptive.coalescePartitions.enabled` configuration?)
> (!hasReducingOperators && maxStageSize > shufflePartitions *
smallPartitionSizeThreshold) ||
Currently, only the rebalance and join operators split skewed partitions. If
there are skewed partitions upstream and they are not rebalance/join operators,
will this lead to excessively large partitions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
