pepijnve commented on issue #23447:
URL: https://github.com/apache/datafusion/issues/23447#issuecomment-4940463355
The physical query plan is
```
AgregateExec: mode=FinalPartitioned, gby=[dst@0 as dst],
aggr=[sum(?table?.value)]
RepartitionExec: partitioning=Hash([dst@0], 8), input_partitions=8
AggregateExec: mode=Partial, gby=[dst@0 as dst],
aggr=[sum(?table?.value)]
ProjectionExec: expr=[dst@3 as dst, value@1 as value]
SortMergeJoinExec: join_type=Inner, on=[(id@0, src@0)]
SortExec: expr=[id@0 ASC], preserve_partitioning=[true]
RepartitionExec: partitioning=Hash([id@0], 8), input_partitions=8
DataSourceExec: file_groups=...
SortExec: expr=[src@0 ASC], preserve_partitioning=[true]
RepartitionExec: partitioning=Hash([src@0], 8),
input_partitions=8
DataSourceExec: file_groups=...
```
tracing the recordbatch streams I can see that all 8 partitions up to the
`ProjectionExec` run until they produce `None` so the problem doesn't seem to
be situated in the sort merge join. When things get stuck, what I'm seeing is
that the `Repartition` streams are returning `Pending`, the final `Aggregate`
returns `Pending` as well, and then everything stops. Is there a code path
where a waker is not being registered when returning `Pending`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
