That's what great about Spark, the community is so active! :) I compiled Mesos 0.20.1 from the source tarball.
Using the Mapr3 Spark 1.1.0 distribution from the Spark downloads page (spark-1.1.0-bin-mapr3.tgz). I see no problems for the workloads we are trying. However, the cluster is small (less than 100 cores across 3 nodes). The workloads reads in just a few gigabytes from HDFS, via an ipython notebook spark shell. thanks, Fi Fairiz "Fi" Azizi On Mon, Oct 6, 2014 at 9:20 AM, Timothy Chen <tnac...@gmail.com> wrote: > Ok I created SPARK-3817 to track this, will try to repro it as well. > > Tim > > On Mon, Oct 6, 2014 at 6:08 AM, RJ Nowling <rnowl...@gmail.com> wrote: > > I've recently run into this issue as well. I get it from running Spark > > examples such as log query. Maybe that'll help reproduce the issue. > > > > > > On Monday, October 6, 2014, Gurvinder Singh <gurvinder.si...@uninett.no> > > wrote: > >> > >> The issue does not occur if the task at hand has small number of map > >> tasks. I have a task which has 978 map tasks and I see this error as > >> > >> 14/10/06 09:34:40 ERROR BlockManagerMasterActor: Got two different block > >> manager registrations on 20140711-081617-711206558-5050-2543-5 > >> > >> Here is the log from the mesos-slave where this container was running. > >> > >> http://pastebin.com/Q1Cuzm6Q > >> > >> If you look for the code from where error produced by spark, you will > >> see that it simply exit and saying in comments "this should never > >> happen, lets just quit" :-) > >> > >> - Gurvinder > >> On 10/06/2014 09:30 AM, Timothy Chen wrote: > >> > (Hit enter too soon...) > >> > > >> > What is your setup and steps to repro this? > >> > > >> > Tim > >> > > >> > On Mon, Oct 6, 2014 at 12:30 AM, Timothy Chen <tnac...@gmail.com> > wrote: > >> >> Hi Gurvinder, > >> >> > >> >> I tried fine grain mode before and didn't get into that problem. > >> >> > >> >> > >> >> On Sun, Oct 5, 2014 at 11:44 PM, Gurvinder Singh > >> >> <gurvinder.si...@uninett.no> wrote: > >> >>> On 10/06/2014 08:19 AM, Fairiz Azizi wrote: > >> >>>> The Spark online docs indicate that Spark is compatible with Mesos > >> >>>> 0.18.1 > >> >>>> > >> >>>> I've gotten it to work just fine on 0.18.1 and 0.18.2 > >> >>>> > >> >>>> Has anyone tried Spark on a newer version of Mesos, i.e. Mesos > >> >>>> v0.20.0? > >> >>>> > >> >>>> -Fi > >> >>>> > >> >>> Yeah we are using Spark 1.1.0 with Mesos 0.20.1. It runs fine in > >> >>> coarse > >> >>> mode, in fine grain mode there is an issue with blockmanager names > >> >>> conflict. I have been waiting for it to be fixed but it is still > >> >>> there. > >> >>> > >> >>> -Gurvinder > >> >>> > >> >>> > --------------------------------------------------------------------- > >> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > >> >>> For additional commands, e-mail: dev-h...@spark.apache.org > >> >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > >> For additional commands, e-mail: dev-h...@spark.apache.org > >> > > > > > > -- > > em rnowl...@gmail.com > > c 954.496.2314 >
