Re: [DataFusion] Blocking async of async is not async

Hi! Thanks for your insights! @andrew the gist I sent does start two runtimes and works. The constraint seems to be that you cannot start a new runtime (or block on one) in the async thread pool, but it seems legal to start a new on

Re: [DataFusion] Blocking async of async is not async

About the questions; - Does anybody know a way to make such a setup work? I have written thin wrapper using async_trait around the parquet reader in my current project, instead of I am writing it, if we have exposed that from parquet and push chunks via channels it would work at the consumer side

Re: [DataFusion] Blocking async of async is not async

Hi, Just a small contribution. We have made some adjustments at Signavio with Jörn to make the chunk reading from S3 streaming at least. You can find the code here. Mind that for this to make a bigger effect increase the hyper's buffer size in its' configuration (which is not included in this code

Re: [DataFusion] Blocking async of async is not async

My understanding of tokio is that there is exactly one global Runtime which has two thread pools: one for synchronous tasks and one for async tasks I am fairly sure there can be only one global Runtime (because when I tried try to exp

Re: [DataFusion] Blocking async of async is not async

Hi Andrew! Thanks for your quick response and sorry it took me so long to answer back. `spawn_blocking` solves the issue: https://gist.github.com/rdettai/d2f9bc59b31785c35dce792878976a19 I am still worried by the amount of thread pools and complexity it creates (1 pool for the outer runtime, 1 p

Re: [DataFusion] Blocking async of async is not async

Tokio has a function `spawn_blocking` that allows running synchronous / blocking code as a future on the current runtime. You can finagle pretty much any combination of sync / async using spawn_blocking and channels, though the resulti