Re: Joining the spark dev community

2014-10-19 Thread Henry Saputra
Hi Saurabh, Good way to start is to use Spark with your applications and file issues you might have found and maybe provide patch for those or existing ones. Please take a look at Spark's how to contribute page [1] to help you get started. Hope this helps. - Henry [1] https://cwiki.apache.org

Re: Raise Java dependency from 6 to 7

2014-10-19 Thread Matei Zaharia
So from my point of view, I'd do it maybe 1-2 years after all the major Hadoop vendors have stopped supporting Java 6. We're not there yet, but we will be soon. The reason is that the cost of staying on Java 6 is much smaller to us (as developers) than the cost of fragmenting the Spark community

Re: Submissions open for Spark Summit East 2015

2014-10-19 Thread Matei Zaharia
BTW several people asked about registration and student passes. Registration will open in a few weeks, and like in previous Spark Summits, I expect there to be a special pass for students. Matei > On Oct 18, 2014, at 9:52 PM, Matei Zaharia wrote: > > After successful events in the past two ye

Re: Raise Java dependency from 6 to 7

2014-10-19 Thread Corey Nolet
A concrete plan and a definite version upon which the upgrade would be applied sounds like it would benefit the community. If you plan far enough out (as Hadoop has done) and give the community enough of a notice, I can't see it being a problem as they would have ample time upgrade. On Sat, Oct

Re: Oryx + Spark mllib

2014-10-19 Thread Nick Pentreath
The shared-nothing load-balanced server architecture works for all but the most massive models - and even then a few big EC2 r3 instances should do the trick. One nice thing about Akka (and especially the new HTTP) is fault tolerance, recovery and potential for persistence. For us arguably the sh

Re: Oryx + Spark mllib

2014-10-19 Thread Sean Owen
Briefly, re: Oryx2, since the intent is for users to write their own serving apps, I though JAX-RS would be more familiar to more developers. I don't know how hard/easy REST APIs are in JAX-RS vs anything else but I suspect it's not much different. The interesting design decision that impacts scal

Re: Oryx + Spark mllib

2014-10-19 Thread Nick Pentreath
Well, when I started development ~2 years ago, Scalatra just appealed more, being more lightweight (I didn't need MVC just barebones REST endpoints), and I still find its API / DSL much nicer to work with. Also, the swagger API docs integration was important to me. So it's more familiarity than any

Re: Oryx + Spark mllib

2014-10-19 Thread Debasish Das
Hi Nick, Any specific reason of choosing scalatra and not play/spray (now that they are getting integrated) ? Sean, Would you be interested in a play and akka clustering based module in oryx2 and see how it compares against the servlets ? I am interested to understand the scalability Thanks