Here's the Weld paper: http://cidrdb.org/cidr2017/papers/p127-palkar-cidr17.pdf
Julian On Thu, Dec 15, 2016 at 2:06 PM, Wes McKinney <wesmck...@gmail.com> wrote: > Sounds great, thanks. > > A number of people are looking at using Arrow for faster serialization > for PySpark, so now that the Java and C++ libraries are compatible > (cf: integration tests) we can make this a reality. > > On Thu, Dec 15, 2016 at 4:54 PM, Julien Le Dem <jul...@dremio.com> wrote: >> I'm happy to reach out to Matei. >> Reynold is on this list and the Arrow PMC as well. >> Wes, I can start with an email and CC you. >> >> >> >> On Thu, Dec 15, 2016 at 11:03 AM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> >>> I already made sure that Matei is aware of this thread. He seemed >>> interested in talking with key Arrow developers. >>> >>> On Thu, Dec 15, 2016 at 10:49 AM, Julian Hyde <jh...@apache.org> wrote: >>> >>> > I think someone should reach out to Matei and Shoumik, and see if they >>> > would like to collaborate. Wes, would you like to do that? >>> > >>> > Also, reach out to the Spark community. Are they aware of Arrow? Are they >>> > planning to use it, or are they developing an alternative? >>> > >>> > Julian >>> > >>> > >>> > >>> > > On Dec 13, 2016, at 10:23 AM, Wes McKinney <wesmck...@gmail.com> >>> wrote: >>> > > >>> > > Looks like there will be a talk about Weld at Strata next Spring: >>> > > >>> > > http://conferences.oreilly.com/strata/strata-ca/public/ >>> > schedule/detail/57646 >>> > > >>> > > It would be nice to see if there's any work that we can be doing in >>> > > Arrow to try to collaborate on such closely-related efforts. >>> > > >>> > > - Wes >>> > > >>> > > On Sat, Nov 26, 2016 at 3:53 PM, Wes McKinney <wesmck...@gmail.com> >>> > wrote: >>> > >> I've always contended that building a sort of "runtime" for in-memory >>> > >> column expressions was a natural next step after hardening Arrow as a >>> > >> data structure and memory exchange mechanism. I'm hopeful that we'll >>> > >> see some cross-pollination between projects like Weld and Arrow to >>> > >> produce open source solutions that help drive more collaboration and >>> > >> consolidation around shared low-level needs for building analytical >>> > >> systems. We still have significant work to do to solidify Arrow as a >>> > >> standard -- anyone all reading can do to help work against >>> > >> fragmentation in data structures / memory representations would be >>> > >> very much appreciated! >>> > >> >>> > >> - Wes >>> > >> >>> > >> On Wed, Nov 23, 2016 at 11:35 AM, Donald Foss <donald.f...@gmail.com> >>> > wrote: >>> > >>> I had that in my queue, and your watching and thoughts about it mean >>> I >>> > need >>> > >>> to watch it today ;) >>> > >>> >>> > >>> Thanks for the link to where you've played with abstraction before. >>> I'm >>> > >>> going to fork a copy and do some playing myself. I'm very interested >>> > in how >>> > >>> universal it can be, and a library to go from expression to universal >>> > >>> intermediate, them output the code in the language of one's choice. >>> The >>> > >>> optimizations could be performed in the intermediate layer, the >>> output >>> > >>> later, or both if there are specific code optimizations available >>> after >>> > >>> translation. >>> > >>> >>> > >>> -Donald >>> > >>> >>> > >>> On Nov 22, 2016 2:31 PM, Julien Le Dem >>> > >>> wrote: >>> > >>> > >>> >> >> >> >> -- >> Julien
