On Mon, Mar 4, 2019 at 5:55 AM Andy Grove <andygrov...@gmail.com> wrote:
> I have been working on a PoC of parallel query execution and it is working > well, and I am now starting to create PRs for the various refactors > necessary for this in DataFusion. > > I haven't been following the async/await and futures/tokio developments > lately but for the PoC I used tokio-threadpool which seems simple to use. > I just wanted to give everyone a chance to give their thoughts on this > before I get too far with my batch of PRs. Is anyone opposed to using > tokio-threadpool? > DataFusion's tasks should be CPU bound and according to tokio-threadpool's documentation [1], it is more suitable for event loops: "It is optimized for the primary Tokio use case of many independent tasks with limited computation and with most tasks waiting on I/O." Rayon seems to follow different semantics, but depending on futures-rs is considerable, especially because it is maintained by the rust lang nursery. [1] https://docs.rs/tokio-threadpool/0.1.12/tokio_threadpool/ Cheers, Krisztian
