andygrove commented on code in PR #2685: URL: https://github.com/apache/datafusion-ballista/pull/2685#discussion_r4408532923
########## ballista/scheduler/src/state/pipelined_join_selection.rs: ########## @@ -0,0 +1,521 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Size-aware join selection for pipelined jobs. +//! +//! A pipelined job is planned with the static planner, so AQE never chooses +//! its join strategies. The scheduler plans it with hash joins, and +//! [`PipelinedJoinSelection`] then turns every `Partitioned` hash join whose +//! build side is not known to fit +//! `ballista.optimizer.hash_join_max_build_partition_bytes` per partition back +//! into a `SortMergeJoinExec`, which can spill. Small build sides stay hash +//! joins, and the distributed planner promotes those under the broadcast +//! threshold to `CollectLeft` as usual. +//! +//! The rule runs among the physical optimizer rules, right after +//! DataFusion's `JoinSelection`, so that the later rules sort and partition +//! the inputs of the sort-merge joins it creates. Rewriting a finished plan +//! instead would lose orderings that a parent relied on: a hash join keeps +//! its probe side's order, so the planner may have dropped a sort above it. +//! +//! This is stricter than AQE, which keeps a hash join when the size is +//! unknown: a pipelined job holds every hash table of the job at once, a hash Review Comment: Filed the job-wide budget follow-up as #2690. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
