Re: Weld

Here's the Weld paper: http://cidrdb.org/cidr2017/papers/p127-palkar-cidr17.pdf Julian On Thu, Dec 15, 2016 at 2:06 PM, Wes McKinney wrote: > Sounds great, thanks. > > A number of people are looking at using Arrow for faster serialization > for PySpark, so now that the Java and C++ libraries ar

Re: Weld

Sounds great, thanks. A number of people are looking at using Arrow for faster serialization for PySpark, so now that the Java and C++ libraries are compatible (cf: integration tests) we can make this a reality. On Thu, Dec 15, 2016 at 4:54 PM, Julien Le Dem wrote: > I'm happy to reach out to Ma

Re: Weld

I'm happy to reach out to Matei. Reynold is on this list and the Arrow PMC as well. Wes, I can start with an email and CC you. On Thu, Dec 15, 2016 at 11:03 AM, Mark Hamstra wrote: > I already made sure that Matei is aware of this thread. He seemed > interested in talking with key Arrow devel

Re: Weld

I already made sure that Matei is aware of this thread. He seemed interested in talking with key Arrow developers. On Thu, Dec 15, 2016 at 10:49 AM, Julian Hyde wrote: > I think someone should reach out to Matei and Shoumik, and see if they > would like to collaborate. Wes, would you like to do

Re: Weld

The PySpark community is aware of arrow, but certainly more reaching out to the Spark SQL devs could really be beneficial to get us all on the same page :) On Thu, Dec 15, 2016 at 10:49 AM Julian Hyde wrote: > I think someone should reach out to Matei and Shoumik, and see if they > would like to

Re: Weld

I think someone should reach out to Matei and Shoumik, and see if they would like to collaborate. Wes, would you like to do that? Also, reach out to the Spark community. Are they aware of Arrow? Are they planning to use it, or are they developing an alternative? Julian > On Dec 13, 2016, at

Re: Weld

Looks like there will be a talk about Weld at Strata next Spring: http://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/57646 It would be nice to see if there's any work that we can be doing in Arrow to try to collaborate on such closely-related efforts. - Wes On Sat, Nov 26, 2

Re: Weld

I've always contended that building a sort of "runtime" for in-memory column expressions was a natural next step after hardening Arrow as a data structure and memory exchange mechanism. I'm hopeful that we'll see some cross-pollination between projects like Weld and Arrow to produce open source sol

Re: Weld

I had that in my queue, and your watching and thoughts about it mean I need to watch it today ;) Thanks for the link to where you've played with abstraction before. I'm going to fork a copy and do some playing myself. I'm very interested in how universal it can be, and a library to go from express

Re: Weld

I finally got time to watch the video. In Arrow talks, I've been selling a similar idea of universal and very efficient UDFs that would use arrow as an interchange. One killer application we talked about with Wes is to have PySpark arrow-enabled. I suspect that the Weld representation would not be

Re: Weld

Looks interesting. I see some commonalities. I hope the original work (in progress?) will make references to Arrow so that we will all know the distinguishing points better. 2016-11-20 8:31 GMT-08:00 Donald E. Foss : > Thanks Julian. Sounds worth a listen. > > Donald E. Foss (mobile-US ET) > > >

Re: Weld

Thanks Julian. Sounds worth a listen. Donald E. Foss (mobile-US ET) > On Nov 19, 2016, at 1:48 PM, Julian Hyde wrote: > > Matei Zaharia just spoke at the AMPlab seminar [1], and showed a couple of > slides about Weld. In the video of the day [2], his talk starts at 4:05:00, > and he starts t

Re: Weld

Matei Zaharia just spoke at the AMPlab seminar [1], and showed a couple of slides about Weld. In the video of the day [2], his talk starts at 4:05:00, and he starts talking about Weld at 4:28:30. The essence is an intermediate language for row-level expressions, with the ability to do limited i

Re: Weld

Did you find that at https://cs.stanford.edu/~matei/? That’s the only thing I can find via Google about it. Do you have more detail or a link to the paper itself? I get the feeling that it is not yet fully complete despite 21 November camera-ready CIDR 2017