devanbenz commented on code in PR #26249:
URL: https://github.com/apache/datafusion/pull/26249#discussion_r4300362066
##########
datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs:
##########
@@ -458,6 +456,209 @@ impl PhysicalExpr for HashTableLookupExpr {
}
}
+/// Build-side lookup for one partition of a `Partitioned` hash join.
+pub(crate) enum PartitionLookup {
+ /// The partition's hash table; probe rows routed here are looked up in it.
+ Map(Arc<Map>),
+ /// The partition is known empty (`false`) or its contents are unknown
(`true`).
+ Const(bool),
+}
+
+/// Routes each probe row to the hash table of the build partition that would
+/// contain it and looks the row up there.
+///
+/// This replaces `CASE hash(keys) % n WHEN i THEN lookup_i ... END` for
+/// partitioned joins. `CaseExpr` evaluates each branch by filtering the batch
+/// down to that branch's rows and merging the partial results, which copies
+/// every column of the batch once per branch. This expression computes the
+/// routing and lookup hashes once, buckets row indices by partition and only
+/// gathers the 8-byte hashes, so the batch columns are never copied.
+pub(crate) struct PartitionedLookupExpr {
+ on_columns: Vec<PhysicalExprRef>,
+ routing_state: SeededRandomState,
+ lookup_state: SeededRandomState,
+ partitions: Vec<PartitionLookup>,
Review Comment:
Also see:
https://github.com/apache/datafusion/pull/26249#issuecomment-6201620853 where
my operator failed to prune while `CASE` did it as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
