Hi Max, I have been experimenting with an async record batch reader and was able to get a working version, but I had to use channels to communicate with the parquet reader, which ran on its own thread.
I have taken a step back now that I have some experience of this and look forward to working with others on a design for async. I know that there are some other contributors who are interested in this as well. I can take an action to start a Google doc where we can discuss this. I think this will be useful while we all align on what problems we are trying to solve and then we can take the findings back to JIRA. I'll create a doc and reply to this thread in the next few days. Here's the JIRA issue that Mahmut was referring to: https://issues.apache.org/jira/browse/ARROW-9275 There are some related issues as well, but without much detail: https://issues.apache.org/jira/browse/ARROW-9674 https://issues.apache.org/jira/browse/ARROW-9754 Thanks, Andy. On Wed, Aug 26, 2020 at 10:31 AM Vertexclique <[email protected]> wrote: > Hi Max; > > There is an open issue in the tracker which needs to gather feedback to > finalize how we will do overall async interface which spans to arrow > crates. Please check that issue, it is mentioning sans IO and several > design considerations. Imo we can carry async discussion under it. > > Best, > Mahmut > > On Aug 26, 2020, 19:18, at 19:18, Max Burke <[email protected]> wrote: > >Out of curiosity, is anyone working on a record batch reader that's > >async > >friendly? > > > >Wanting to know if it's something I could wait on/help out with, or if > >it's > >something we could start working on too. > > > >-- > >-Max >
