Re: Notes on writing complex spark applications

2014-11-24 Thread Evan R. Sparks
Thanks Patrick, You raise a good point - for this to be useful it's imperative that it is updated with new versions of spark. My thought with putting it on the wiki was that it's lower friction for community members to edit, but it likely won't have the same level of quality control as the existi

Re: Notes on writing complex spark applications

2014-11-23 Thread Patrick Wendell
Hey Evan, It might be nice to merge this into existing documentation. In particular, a lot of this could serve to update the current tuning section and programming guides. It could also work to paste this wholesale as a reference for Spark users, but in that case it's less likely to get updated w

Re: Notes on writing complex spark applications

2014-11-23 Thread Inkyu Lee
Very helpful!! thank you very much! 2014-11-24 2:17 GMT+09:00 Sam Bessalah : > Thanks Evan, this is great. > On Nov 23, 2014 5:58 PM, "Evan R. Sparks" wrote: > > > Hi all, > > > > Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been > > working on a short document about writing

Re: Notes on writing complex spark applications

2014-11-23 Thread Sam Bessalah
Thanks Evan, this is great. On Nov 23, 2014 5:58 PM, "Evan R. Sparks" wrote: > Hi all, > > Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been > working on a short document about writing high performance Spark > applications based on our experience developing MLlib, GraphX, ml-m

Re: Notes on writing complex spark applications

2014-11-23 Thread andy petrella
Cool! On Sun Nov 23 2014 at 5:58:03 PM Evan R. Sparks wrote: > Hi all, > > Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been > working on a short document about writing high performance Spark > applications based on our experience developing MLlib, GraphX, ml-matrix, > pipeli

Notes on writing complex spark applications

2014-11-23 Thread Evan R. Sparks
Hi all, Shivaram Venkataraman, Joseph Gonzalez, Tomer Kaftan, and I have been working on a short document about writing high performance Spark applications based on our experience developing MLlib, GraphX, ml-matrix, pipelines, etc. It may be a useful document both for users and new Spark develope