> +1 to what Ian said. I'll also add, as was brought up in the earlier> pandas > API discussion, that a blog post (once you've got something > ready to use) would be a good idea.
That's a great suggestion! I'll keep that in mind as we get closer to having something usable. > I don't fully understand this statement. I suspect you are correct > but, since I don't understand the reasons, I can't really say what we > can do to help. For example, if it's purely the fact that it is an > ASF project then we can make a new arrow-xyz repo (you'd need a > committer to dedicate time to reviews and you would need to convince > several PMC members to commit time for validating your releases). My thought was simply that, if it were easy to build and install the proposed API alongside `libarrow`, then the terseness of the proposed API could make it easier for developers to evaluate Arrow for analytics use-cases. That said, my perspective on this might be skewed because we build all C++ libraries from source in an integration build system at my company (which presents its own challenges), so the initial barrier to adopting a new library is fairly high. If it's normal for C++ developers to use pre-built libraries, then my point is moot. Thank you for teasing this nuance out of me, my original post was not sufficiently clear about this. Either way, I still agree with Ian's point that avoiding consolidation of such functionality in the Arrow repos would be in the best interest of the Arrow ecosystem. From: dev@arrow.apache.org At: 01/27/23 13:06:20 UTC-5:00To: dev@arrow.apache.org Subject: Re: [Discuss] C++ query builder/execution API +1 to what Ian said. I'll also add, as was brought up in the earlier pandas API discussion, that a blog post (once you've got something ready to use) would be a good idea. > On the other hand, making such an interface readily available alongside `libarrow` could increase the adoption of Arrow among certain developers (e.g. in finance/fintech). I don't fully understand this statement. I suspect you are correct but, since I don't understand the reasons, I can't really say what we can do to help. For example, if it's purely the fact that it is an ASF project then we can make a new arrow-xyz repo (you'd need a committer to dedicate time to reviews and you would need to convince several PMC members to commit time for validating your releases). On Fri, Jan 27, 2023 at 9:10 AM Ian Cook <i...@ursacomputing.com> wrote: > > Hi Shoumyo, > > This is exciting—thank you for the thoughtfulness you have put into > this proposal. > > This topic of a C++ dataframe API for Arrow-native engine(s) has come > up in the past [3], but the bulk of the previous discussion about this > predated Substrait. With the Substrait project now quickly gaining > momentum, it seems an excellent time to revisit this topic and to > incorporate Substrait into it, as you have done. > > I strongly believe that this work should happen in a repository that > is outside of the Arrow project. Many of the exciting developments in > Arrow-land these days are happening in the broader ecosystem around > Arrow. The proposed API could be used independently of Arrow libraries > (for example, it could be used with DuckDB). For projects like this, I > think our hope as Arrow maintainers is to "let a hundred flowers > bloom" around Arrow (all with excellent operability based on Arrow > standards) rather than centralizing the work inside Arrow > repositories. We can use resources including the "Powered by Arrow" > and "Powered by Substrait" pages to highlight the project. > > Thank you, > Ian > > [3] https://docs.google.com/document/d/1XHe_j87n2VHGzEbnLe786GHbbcbrzbjgG8D0IXWAeHg/ > [4] https://arrow.apache.org/powered_by/ > [5] https://substrait.io/community/powered_by/ > > On Wed, Jan 25, 2023 at 1:02 PM Shoumyo Chakravorti (BLOOMBERG/ 731 > LEX) <schakravo...@bloomberg.net> wrote: > > > > Hi Arrow developers! > > > > This is my first time posting on this mailing list, so please let me know if this post belongs elsewhere. > > > > I and a few colleagues plan to implement a C++ interface for building read-only queries and executing those against Arrow Datasets through Substrait consumers like Acero, DuckDB, and Velox. Since we hope to build this out in the open, I have outlined the kind of interface that we intend to build in this Google doc [0]. > > > > I'm making this post for a few reasons: > > > > - To gauge whether the community feels like this work would be worth pursuing as an open-source project > > - To receive feedback on the proposed interface and ensure that we would be able to accommodate a wide variety of use-cases (please feel free to leave comments directly on the doc) > > - To connect with developers who might be interested in collaborating on this effort > > > > Relatedly, I would like to get the Arrow developers' thoughts on whether it would make sense to pursue this work as an official Arrow project (e.g. in an experimental repo) or if it would be better as a standalone project. I understand that pursuing this as an Arrow project would have its downsides (like increased review/maintenance burden) and risks confusing new users as to what the official Arrow libraries aim to solve [1]. On the other hand, making such an interface readily available alongside `libarrow` could increase the adoption of Arrow among certain developers (e.g. in finance/fintech). Regardless of your opinion, I'd love to hear your thoughts on which approach makes more sense. > > > > Please feel free to reply here on the mailing list or leave comments on the linked Google Doc! > > > > [0]: https://docs.google.com/document/d/1_ktKxtOFW1grD-VcbBNc0FaP4g5j7vSx9bO2ht59JFA > > [1]: https://www.datawill.io/posts/apache-arrow-2022-reflection/#who-is-libarrows-and -aceros-audience
