Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

2014-03-31 Thread Andrew Or
+1 tested on OSX On Mon, Mar 31, 2014 at 4:33 PM, Kevin Markey wrote: > I had specifically requested that the ASM shading be included in the RC, > hence my testing focused on that, but I ran other tests as well. Tested > with a build of our project, running one of our applications from that > b

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

2014-03-31 Thread Kevin Markey
I had specifically requested that the ASM shading be included in the RC, hence my testing focused on that, but I ran other tests as well. Tested with a build of our project, running one of our applications from that build in yarn-standalone on a pseudocluster, and successfully redeploying and

Re: Calling Spark enthusiasts in NYC

2014-03-31 Thread shenyan zhen
I'll be very interested. Quick intro of myself: code java during the day, code Scala during the night. On Mar 31, 2014 1:23 PM, "Andy Konwinski" wrote: > Hi folks, > > We have seen a lot of community growth outside of the Bay Area and we are > looking to help spur even more! > > For starters, th

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-31 Thread Debasish Das
Thanks for the pointers. That's the plan...I am going to have default ALS, variable projection at each ALS step, and then using the constrained first order solver that you have... I am hoping ECOS QP will beat all of them...I don't know...some of these obvious assumptions does not work when the d

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

2014-03-31 Thread Tathagata Das
Yes, lets extend the vote for two more days from now. So the vote is open till *Wednesday, April 02, at 20:00 UTC* On that note, my +1 TD On Mon, Mar 31, 2014 at 9:57 AM, Patrick Wendell wrote: > Yeah good point. Let's just extend this vote another few days? > > > On Mon, Mar 31, 2014 at 8:

Re: Calling Spark enthusiasts in NYC

2014-03-31 Thread Surendranauth Hiraman
We at Velos are willing to get involved too. Please let me know. -Suren SUREN HIRAMAN, VP TECHNOLOGY Velos Accelerating Machine Learning 440 NINTH AVENUE, 11TH FLOOR NEW YORK, NY 10001 O: (917) 525-2466 ext. 105 F: 646.349.4063 E: suren.hiraman@v elos.io W: www.velos.io On Mon, Mar 31, 2014

Re: Calling Spark enthusiasts in NYC

2014-03-31 Thread Krakna H
Would love to see this happen and happy to help set this up in NYC. On Mon, Mar 31, 2014 at 1:52 PM, andy [via Apache Spark Developers List] < ml-node+s1001551n6167...@n3.nabble.com> wrote: > Responses about London, Montreal/Toronto, DC, Chicago. Great coverage so > far, and keep 'em coming! (st

Re: Calling Spark enthusiasts in NYC

2014-03-31 Thread Andy Konwinski
Responses about London, Montreal/Toronto, DC, Chicago. Great coverage so far, and keep 'em coming! (still looking for an NYC connection) I'll reply to each of you off-list to coordinate next-steps for setting up a Spark meetup in your home area. Thanks again, this is super exciting. Andy On Mo

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-31 Thread David Hall
On Mon, Mar 31, 2014 at 10:15 AM, Debasish Das wrote: > I added eclipse support in my qp branch: > > https://github.com/debasish83/breeze/tree/qp Ok, great. Totally fine for this to come in with the QP. > > > For the QP solver I will look into this solver http://www.joptimizer.com/ > > Right n

Calling Spark enthusiasts in NYC

2014-03-31 Thread Andy Konwinski
Hi folks, We have seen a lot of community growth outside of the Bay Area and we are looking to help spur even more! For starters, the organizers of the Spark meetups here in the Bay Area want to help anybody that is interested in setting up a meetup in a new city. Some amazing Spark champions ha

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-31 Thread Debasish Das
I added eclipse support in my qp branch: https://github.com/debasish83/breeze/tree/qp For the QP solver I will look into this solver http://www.joptimizer.com/ Right now my plan is to use Professor Boyd's ECOS solver which is also designed in the very similar lines but has been tested to solve e

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

2014-03-31 Thread Patrick Wendell
Yeah good point. Let's just extend this vote another few days? On Mon, Mar 31, 2014 at 8:12 AM, Tom Graves wrote: > I should probably pull this off into another thread, but going forward can > we try to not have the release votes end on a weekend? Since we only seem > to give 3 days, it makes i

Re: The difference between driver and master in Spark

2014-03-31 Thread Patrick Wendell
Checkout this page: http://spark.incubator.apache.org/docs/latest/cluster-overview.html On Mon, Mar 31, 2014 at 9:11 AM, Nan Zhu wrote: > master is managing the resources in the cluster, e.g. ensuring all > components can work together, master/worker/driver > > e.g. you have to submit your appl

Re: The difference between driver and master in Spark

2014-03-31 Thread Nan Zhu
master is managing the resources in the cluster, e.g. ensuring all components can work together, master/worker/driver e.g. you have to submit your application with the path: driver -> master -> worker then the driver take most of the responsibility of running your application, e.g. scheduli

Re: The difference between driver and master Spark

2014-03-31 Thread Andy Konwinski
The master is what monitors and manages jobs. It can be a master in a Spark standalone scheduler (which itself might be running inside of YARN) or a Mesos master. The driver is the program that the user interfaces with to talk to their Spark job. That communication is primarily dive through a Spar

Re: [VOTE] Release Apache Spark 0.9.1 (RC3)

2014-03-31 Thread Tom Graves
I should probably pull this off into another thread, but going forward can we try to not have the release votes end on a weekend? Since we only seem to give 3 days, it makes it really hard for anyone who is offline for the weekend to try it out.   Either that or extend the voting for more then 3

The difference between driver and master in Spark

2014-03-31 Thread Dan
Hi, I've been recently reading spark code and confused about driver and master. What's the difference between them? When I run spark in standalone cluster, from the log it seems that the driver has not been launched. Thanks, Dan -- View this message in context: http://apache-spark-developer

Re: Error executing sql using shark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1

2014-03-31 Thread qingyang li
if i delete protobuf-java-2.4.1-shaded.jar , there will be other errors. 2014-03-28 5:25 GMT+08:00 yao : > So it looks like shark introduces protobuf-2.4, are you sure shark depends > on the right spark version ? I am not familiar with shark's dependencies, > but in your case, you could delete t