comphead commented on code in PR #12082:
URL: https://github.com/apache/datafusion/pull/12082#discussion_r1723580882
##########
datafusion/physical-plan/src/joins/sort_merge_join.rs:
##########
@@ -1356,16 +1392,82 @@ impl SMJStream {
pre_mask.clone()
};
+ // Try to calculate if the buffered batch we scan is the
last one for specific stream row and join key
+ // for Batchsize == 1
self.buffered_data.scanning_finished() works well
+ // For other scenarios its an attempt to figure out there
is no more rows matching the same join key
+ let last_batch = if self.batch_size == 1 {
Review Comment:
@korowa @viirya @alamb
Hi guys appreciate if you have any other ideas how to calculate the last
batch.
AntiJoin relies exactly on having the last batch, to calculate the predicate
for join key correctly.
I'm trying to figure out there is no more buffered rows incoming for the
given streaming join key. The approach is still not perfect as it still allows
the tests to fail time to time although it becomes more stable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
