englefly commented on code in PR #27259:
URL: https://github.com/apache/doris/pull/27259#discussion_r1399931982
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java:
##########
@@ -299,12 +303,41 @@ public Cost visitPhysicalHashJoin(
0
);
}
- return CostV1.of(context.getSessionVariable(), leftRowCount +
rightRowCount + outputRowCount,
+ return CostV1.of(context.getSessionVariable(),
+ (leftRowCount + rightRowCount + outputRowCount)
+ * skewPenalty(physicalHashJoin, probeStats,
buildStats, totalInstanceNumber),
rightRowCount,
0
);
}
+ private double skewPenalty(PhysicalHashJoin<? extends Plan, ? extends
Plan> physicalHashJoin,
+ Statistics probeStats,
+ Statistics buildStats,
+ int totalInstanceNumber) {
+ double penalty = 1.0;
+ for (Expression conj : physicalHashJoin.getHashJoinConjuncts()) {
+ EqualTo eq = (EqualTo) JoinUtils.swapEqualToForChildrenOrder(
+ (EqualTo) conj, physicalHashJoin.left().getOutputSet());
+ ColumnStatistic leftColStats =
probeStats.findColumnStatistics(eq.left());
+ if (leftColStats != null && !leftColStats.isUnKnown()) {
+ if (leftColStats.ndv < beNumber) {
+ penalty = totalInstanceNumber / Math.max(1.0,
leftColStats.ndv);
Review Comment:
1. optimizer does not know the number of instances assigned to this
PlanNode. totalInstanceNumber is the upper bound.
2. it is more important to balance workload between BEs than that of between
instances, that is why penalty is triggered when "ndv < beNumber"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
