Hi Shoumyo, This is exciting—thank you for the thoughtfulness you have put into this proposal.
This topic of a C++ dataframe API for Arrow-native engine(s) has come up in the past [3], but the bulk of the previous discussion about this predated Substrait. With the Substrait project now quickly gaining momentum, it seems an excellent time to revisit this topic and to incorporate Substrait into it, as you have done. I strongly believe that this work should happen in a repository that is outside of the Arrow project. Many of the exciting developments in Arrow-land these days are happening in the broader ecosystem around Arrow. The proposed API could be used independently of Arrow libraries (for example, it could be used with DuckDB). For projects like this, I think our hope as Arrow maintainers is to "let a hundred flowers bloom" around Arrow (all with excellent operability based on Arrow standards) rather than centralizing the work inside Arrow repositories. We can use resources including the "Powered by Arrow" and "Powered by Substrait" pages to highlight the project. Thank you, Ian [3] https://docs.google.com/document/d/1XHe_j87n2VHGzEbnLe786GHbbcbrzbjgG8D0IXWAeHg/ [4] https://arrow.apache.org/powered_by/ [5] https://substrait.io/community/powered_by/ On Wed, Jan 25, 2023 at 1:02 PM Shoumyo Chakravorti (BLOOMBERG/ 731 LEX) <schakravo...@bloomberg.net> wrote: > > Hi Arrow developers! > > This is my first time posting on this mailing list, so please let me know if > this post belongs elsewhere. > > I and a few colleagues plan to implement a C++ interface for building > read-only queries and executing those against Arrow Datasets through > Substrait consumers like Acero, DuckDB, and Velox. Since we hope to build > this out in the open, I have outlined the kind of interface that we intend to > build in this Google doc [0]. > > I'm making this post for a few reasons: > > - To gauge whether the community feels like this work would be worth > pursuing as an open-source project > - To receive feedback on the proposed interface and ensure that we would > be able to accommodate a wide variety of use-cases (please feel free to leave > comments directly on the doc) > - To connect with developers who might be interested in collaborating on > this effort > > Relatedly, I would like to get the Arrow developers' thoughts on whether it > would make sense to pursue this work as an official Arrow project (e.g. in an > experimental repo) or if it would be better as a standalone project. I > understand that pursuing this as an Arrow project would have its downsides > (like increased review/maintenance burden) and risks confusing new users as > to what the official Arrow libraries aim to solve [1]. On the other hand, > making such an interface readily available alongside `libarrow` could > increase the adoption of Arrow among certain developers (e.g. in > finance/fintech). Regardless of your opinion, I'd love to hear your thoughts > on which approach makes more sense. > > Please feel free to reply here on the mailing list or leave comments on the > linked Google Doc! > > [0]: > https://docs.google.com/document/d/1_ktKxtOFW1grD-VcbBNc0FaP4g5j7vSx9bO2ht59JFA > [1]: > https://www.datawill.io/posts/apache-arrow-2022-reflection/#who-is-libarrows-and-aceros-audience
