I was able to reproduce it on a small 4 node cluster (1 mesos master and 3 mesos slaves) with relatively low-end specs. As I said, I just ran the log query examples with the fine-grained mesos mode.
Spark 1.1.0 and mesos 0.20.1. Fairiz, could you try running the logquery example included with Spark and see what you get? Thanks! On Mon, Oct 6, 2014 at 8:07 PM, Fairiz Azizi <code...@gmail.com> wrote: > That's what great about Spark, the community is so active! :) > > I compiled Mesos 0.20.1 from the source tarball. > > Using the Mapr3 Spark 1.1.0 distribution from the Spark downloads page > (spark-1.1.0-bin-mapr3.tgz). > > I see no problems for the workloads we are trying. > > However, the cluster is small (less than 100 cores across 3 nodes). > > The workloads reads in just a few gigabytes from HDFS, via an ipython > notebook spark shell. > > thanks, > Fi > > > > Fairiz "Fi" Azizi > > On Mon, Oct 6, 2014 at 9:20 AM, Timothy Chen <tnac...@gmail.com> wrote: > >> Ok I created SPARK-3817 to track this, will try to repro it as well. >> >> Tim >> >> On Mon, Oct 6, 2014 at 6:08 AM, RJ Nowling <rnowl...@gmail.com> wrote: >> > I've recently run into this issue as well. I get it from running Spark >> > examples such as log query. Maybe that'll help reproduce the issue. >> > >> > >> > On Monday, October 6, 2014, Gurvinder Singh <gurvinder.si...@uninett.no >> > >> > wrote: >> >> >> >> The issue does not occur if the task at hand has small number of map >> >> tasks. I have a task which has 978 map tasks and I see this error as >> >> >> >> 14/10/06 09:34:40 ERROR BlockManagerMasterActor: Got two different >> block >> >> manager registrations on 20140711-081617-711206558-5050-2543-5 >> >> >> >> Here is the log from the mesos-slave where this container was running. >> >> >> >> http://pastebin.com/Q1Cuzm6Q >> >> >> >> If you look for the code from where error produced by spark, you will >> >> see that it simply exit and saying in comments "this should never >> >> happen, lets just quit" :-) >> >> >> >> - Gurvinder >> >> On 10/06/2014 09:30 AM, Timothy Chen wrote: >> >> > (Hit enter too soon...) >> >> > >> >> > What is your setup and steps to repro this? >> >> > >> >> > Tim >> >> > >> >> > On Mon, Oct 6, 2014 at 12:30 AM, Timothy Chen <tnac...@gmail.com> >> wrote: >> >> >> Hi Gurvinder, >> >> >> >> >> >> I tried fine grain mode before and didn't get into that problem. >> >> >> >> >> >> >> >> >> On Sun, Oct 5, 2014 at 11:44 PM, Gurvinder Singh >> >> >> <gurvinder.si...@uninett.no> wrote: >> >> >>> On 10/06/2014 08:19 AM, Fairiz Azizi wrote: >> >> >>>> The Spark online docs indicate that Spark is compatible with Mesos >> >> >>>> 0.18.1 >> >> >>>> >> >> >>>> I've gotten it to work just fine on 0.18.1 and 0.18.2 >> >> >>>> >> >> >>>> Has anyone tried Spark on a newer version of Mesos, i.e. Mesos >> >> >>>> v0.20.0? >> >> >>>> >> >> >>>> -Fi >> >> >>>> >> >> >>> Yeah we are using Spark 1.1.0 with Mesos 0.20.1. It runs fine in >> >> >>> coarse >> >> >>> mode, in fine grain mode there is an issue with blockmanager names >> >> >>> conflict. I have been waiting for it to be fixed but it is still >> >> >>> there. >> >> >>> >> >> >>> -Gurvinder >> >> >>> >> >> >>> >> --------------------------------------------------------------------- >> >> >>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> >>> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >>> >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >> > >> > >> > -- >> > em rnowl...@gmail.com >> > c 954.496.2314 >> > > -- em rnowl...@gmail.com c 954.496.2314
