andygrove commented on code in PR #2900:
URL: 
https://github.com/apache/datafusion-ballista/pull/2900#discussion_r4643004577


##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -387,15 +500,26 @@ impl Stream for ColumnStatsStream {
         let this = self.get_mut();
         let poll = this.input.poll_next_unpin(cx);
         if let Poll::Ready(Some(Ok(batch))) = &poll {
-            let counted = batch
+            let counted: Vec<&ArrayRef> = batch
                 .columns()
                 .iter()
                 .enumerate()
                 .filter(|(i, _)| Some(*i) != this.ignored_column)
-                .map(|(_, column)| column);
-            for (slot, column) in this.null_counts.iter_mut().zip(counted) {
+                .map(|(_, column)| column)
+                .collect();
+            for (slot, column) in this.null_counts.iter_mut().zip(&counted) {
                 *slot += column.logical_null_count() as u64;
             }
+            for (slot, column) in this.min_max.iter_mut().zip(&counted) {
+                if let Some((min, max)) = slot {
+                    let updated = min
+                        .update_batch(&[Arc::clone(*column)])
+                        .and_then(|_| 
max.update_batch(&[Arc::clone(*column)]));
+                    if let Err(e) = updated {

Review Comment:
   Since the range is only an optimization hint, what do you think about 
dropping it (setting the slot to `None`) when `update_batch` fails, instead of 
failing the task? That would match how `merge_min_max` handles it.



##########
ballista/core/src/execution_plans/shuffle_writer.rs:
##########
@@ -779,6 +906,7 @@ impl ShuffleWriterExec {
             let channel_capacity = config.shuffle_writer_channel_capacity();
             column_stats.reset(
                 plan.schema().fields().len(),
+                &sketched_columns(plan.properties().output_partitioning(), 
true),

Review Comment:
   Should min/max collection be gated by a config key like the distinct-count 
sketches are? Right now `true` is passed unconditionally here and in 
`sort_shuffle/writer.rs`, so the static (non-AQE) path pays for stats nothing 
reads.



##########
ballista/scheduler/src/state/aqe/optimizer_rule/propagate_empty.rs:
##########
@@ -285,6 +289,98 @@ pub fn as_join(plan: &Arc<dyn ExecutionPlan>) -> 
Option<JoinInfo<'_>> {
     None
 }
 
+/// Whether an inner or semi join can't match any row: for some key pair, the
+/// exact ranges measured on the finished shuffles feeding the join don't
+/// overlap.
+fn keys_never_match(plan: &Arc<dyn ExecutionPlan>) -> bool {

Review Comment:
   The unit tests here are thorough. Would you be up for adding an end-to-end 
test too? Something in standalone mode with AQE on, where a join on disjoint 
key ranges returns zero rows and the plan shows `EmptyExec`.



##########
ballista/core/src/execution_plans/sort_shuffle/writer.rs:
##########
@@ -1221,6 +1221,7 @@ async fn run_coordinator(
         .map_or(0, |schema| schema.fields().len());
     writer.column_stats.reset(
         written_columns,
+        &sketched_columns(&writer.routing.output_partitioning(), true),

Review Comment:
   Small naming question. `sketched_columns(.., true)` is reused here to pick 
the min/max columns. Would a separate `hash_key_columns(partitioning)` helper 
read more clearly?



##########
ballista/scheduler/src/state/aqe/optimizer_rule/propagate_empty.rs:
##########
@@ -141,6 +143,8 @@ impl PropagateEmptyExecRule {
             && is_empty_exec!(sort_preserving_merge.input())
         {
             empty_exec!(sort_preserving_merge)
+        } else if keys_never_match(&plan) {

Review Comment:
   Nice that this sits behind `ballista.planner.propagate_empty.enabled`. Could 
the rustdoc for that config mention that it now also covers joins on disjoint 
key ranges?



##########
ballista/scheduler/src/state/execution_stage.rs:
##########
@@ -772,6 +840,15 @@ impl RunningStage {
             for (merged, sketch) in sketches.iter_mut().zip(task_sketches) {
                 merged.add(sketch);
             }
+            for (merged, range) in ranges.iter_mut().zip(task_ranges) {
+                merged.add(range);
+            }
+        }
+        for (slot, range) in output_column_stats.iter_mut().zip(ranges) {
+            if let Some((min, max)) = range.into_range() {
+                slot.min_value = Precision::Exact(min);

Review Comment:
   Question on a side effect. With `Exact` min/max now on `ExchangeExec`, 
DataFusion's join cardinality estimates will start using these ranges. Could 
that shift `SelectJoinRule` build-side choices on queries where no join is 
eliminated? The `EXPLAIN` diff would be a good place to check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to