[
https://issues.apache.org/jira/browse/IMPALA-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012730#comment-18012730
]
ASF subversion and git services commented on IMPALA-14263:
----------------------------------------------------------
Commit cc1cbb559a849bc75b49bc44f4869e8eb8d9e47f in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=cc1cbb559 ]
IMPALA-14263: Add broadcast_cost_scale_factor option
This commit enhances the distributed planner's costing model for
broadcast joins by introducing the `broadcast_cost_scale_factor` query
option. This option enables users to fine-tune the planner's decision
between broadcast and partitioned joins.
Key changes:
- The total broadcast cost is scaled by the new
`broadcast_cost_scale_factor` query option, allowing users to favor or
penalize broadcast joins as needed when setting query hint is not
feasible.
- Updated the planner logic and test cases to reflect the new costing
model and options.
This addresses scenarios where the default costing could lead to
suboptimal join distribution choices, particularly in a large-scale
cluster where the number of executors can increase broadcast cost, while
choosing a partitioned strategy can lead to data skew. Admin can set
`broadcast_cost_scale_factor` less than 1.0 to make DistributedPlanner
favor broadcast more than partitioned join (with possible downside of
higher memory usage per query and higher network transmission).
Existing query hints still take precedence over this option. Note that
this option is applied independent of `broadcast_to_partition_factor`
option (see IMPALA-10287). In MT_DOP>1 setup, it should be sufficient to
set `use_dop_for_costing=True` and tune `broadcast_to_partition_factor`
only.
Testing:
Added FE tests.
Change-Id: I475f8a26b2171e87952b69f66a5c18f77c2b3133
Reviewed-on: http://gerrit.cloudera.org:8080/23258
Reviewed-by: Wenzhe Zhou <[email protected]>
Reviewed-by: Aman Sinha <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Broadcast cost in planner is skewed by the number of nodes comparing to
> partition cost
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-14263
> URL: https://issues.apache.org/jira/browse/IMPALA-14263
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Wenzhe Zhou
> Assignee: Riza Suminto
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> broadCast Cost = dataPayload + hashTblBuildCost = 2 x (rhsDataSize *
> leftChildNodes)
> partition Cost = Math.round(lhsNetworkCost + rhsNetworkCost + rhsDataSize)
> The number of nodes skews broadcast cost on bigger clusters, which makes
> broadcast cost much bigger than partitioned join cost, e.g. planner favor
> partition strategy for big cluster.
> We probably need to introduce new heuristics to join strategy decision, like
> including number of nodes in partitioned join cost model. We also need a way
> to check for the degree of skew on the join key during the planning phase. If
> the skew is on the higher side, we would want to bias the cost model towards
> broadcast.
> Adding join hints in the query is the recommended workaround to force
> broadcast join in the cases where join keys are skewed, especially for larger
> clusters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
