Hey Ian! > This is exciting—thank you for the thoughtfulness you have put into > this proposal. > > This topic of a C++ dataframe API for Arrow-native engine(s) has come > up in the past [3], but the bulk of the previous discussion about this > predated Substrait. With the Substrait project now quickly gaining > momentum, it seems an excellent time to revisit this topic and to > incorporate Substrait into it, as you have done.
Appreciate you taking the time to look over the proposal. Indeed, this project seems like it will be a great fit for Substrait. Seeing as Acero, DuckDB, and Velox all seem to be aiming for compatibility with Substrait, producing Substrait should greatly reduce the development cost of integrating with all of these engines. > I strongly believe that this work should happen in a repository that > is outside of the Arrow project. Many of the exciting developments in > Arrow-land these days are happening in the broader ecosystem around > Arrow. The proposed API could be used independently of Arrow libraries > (for example, it could be used with DuckDB). For projects like this, I > think our hope as Arrow maintainers is to "let a hundred flowers > bloom" around Arrow (all with excellent operability based on Arrow > standards) rather than centralizing the work inside Arrow > repositories. I tend to agree with this line of thought -- since we'll be building on _top_ of open standards (Arrow + Substrait), there's no technical reason to include the proposed interface in the core Arrow libraries. Making this a separate project will also force us to identify gaps in the standards that might hinder interoperability, and addressing those gaps will benefit the ecosystem as a whole. Thanks again for taking a look and sharing your perspective! From: dev@arrow.apache.org At: 01/27/23 12:10:42 UTC-5:00To: dev@arrow.apache.org Subject: Re: [Discuss] C++ query builder/execution API Hi Shoumyo, This is exciting—thank you for the thoughtfulness you have put into this proposal. This topic of a C++ dataframe API for Arrow-native engine(s) has come up in the past [3], but the bulk of the previous discussion about this predated Substrait. With the Substrait project now quickly gaining momentum, it seems an excellent time to revisit this topic and to incorporate Substrait into it, as you have done. I strongly believe that this work should happen in a repository that is outside of the Arrow project. Many of the exciting developments in Arrow-land these days are happening in the broader ecosystem around Arrow. The proposed API could be used independently of Arrow libraries (for example, it could be used with DuckDB). For projects like this, I think our hope as Arrow maintainers is to "let a hundred flowers bloom" around Arrow (all with excellent operability based on Arrow standards) rather than centralizing the work inside Arrow repositories. We can use resources including the "Powered by Arrow" and "Powered by Substrait" pages to highlight the project. Thank you, Ian [3] https://docs.google.com/document/d/1XHe_j87n2VHGzEbnLe786GHbbcbrzbjgG8D0IXWAeHg/ [4] https://arrow.apache.org/powered_by/ [5] https://substrait.io/community/powered_by/ On Wed, Jan 25, 2023 at 1:02 PM Shoumyo Chakravorti (BLOOMBERG/ 731 LEX) <schakravo...@bloomberg.net> wrote: > > Hi Arrow developers! > > This is my first time posting on this mailing list, so please let me know if this post belongs elsewhere. > > I and a few colleagues plan to implement a C++ interface for building read-only queries and executing those against Arrow Datasets through Substrait consumers like Acero, DuckDB, and Velox. Since we hope to build this out in the open, I have outlined the kind of interface that we intend to build in this Google doc [0]. > > I'm making this post for a few reasons: > > - To gauge whether the community feels like this work would be worth pursuing as an open-source project > - To receive feedback on the proposed interface and ensure that we would be able to accommodate a wide variety of use-cases (please feel free to leave comments directly on the doc) > - To connect with developers who might be interested in collaborating on this effort > > Relatedly, I would like to get the Arrow developers' thoughts on whether it would make sense to pursue this work as an official Arrow project (e.g. in an experimental repo) or if it would be better as a standalone project. I understand that pursuing this as an Arrow project would have its downsides (like increased review/maintenance burden) and risks confusing new users as to what the official Arrow libraries aim to solve [1]. On the other hand, making such an interface readily available alongside `libarrow` could increase the adoption of Arrow among certain developers (e.g. in finance/fintech). Regardless of your opinion, I'd love to hear your thoughts on which approach makes more sense. > > Please feel free to reply here on the mailing list or leave comments on the linked Google Doc! > > [0]: https://docs.google.com/document/d/1_ktKxtOFW1grD-VcbBNc0FaP4g5j7vSx9bO2ht59JFA > [1]: https://www.datawill.io/posts/apache-arrow-2022-reflection/#who-is-libarrows-and -aceros-audience
