pepijnve commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2943553646
@zhuqi-lucas FYI, I've reworked the 'infinite stream' to 'range stream' in my tests. It simply emits a `Range<i64>` now. I've added an additional evil test case for sort-merge join where I'm inner joining `[i64::MIN, 0]` with `[0, i64::MAX]` with no further statistics. Again to emulate the join blocking in its data collection phase. @ozankabak I still don't think you need all this API work since there's a zero API change way to deal with cancellation already. Tests all pass with no API changes in the 'all Stream implementations must be well behaved Tokio citizens' approach. I understand the performance concern, but maybe it's a bit premature to design APIs before knowing what the actual performance impact is? In terms of code changes I don't think the complexity argument holds since the required code changes were fairly trivial. I've found a dedicated machine to run the benchmarks on in the meantime. It's 10 year old hardware (Xeon E5620) so the compile times take forever, but should be good enough for relative comparisons. Will post results when I get them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
