The problem is not really for local[1] or local. The problem arises when there are more input streams than there are cores. But I agree, for people who are just beginning to use it by running it locally, there should be a check addressing this.
I made a JIRA for this. https://issues.apache.org/jira/browse/SPARK-2464 TD On Sun, Jul 13, 2014 at 4:26 PM, Sean Owen <so...@cloudera.com> wrote: > How about a PR that rejects a context configured for local or local[1]? As > I understand it is not intended to work and has bitten several people. > On Jul 14, 2014 12:24 AM, "Michael Campbell" <michael.campb...@gmail.com> > wrote: > >> This almost had me not using Spark; I couldn't get any output. It is not >> at all obvious what's going on here to the layman (and to the best of my >> knowledge, not documented anywhere), but now you know you'll be able to >> answer this question for the numerous people that will also have it. >> >> >> On Sun, Jul 13, 2014 at 5:24 PM, Walrus theCat <walrusthe...@gmail.com> >> wrote: >> >>> Great success! >>> >>> I was able to get output to the driver console by changing the >>> construction of the Streaming Spark Context from: >>> >>> val ssc = new StreamingContext("local" /**TODO change once a cluster is >>> up **/, >>> "AppName", Seconds(1)) >>> >>> >>> to: >>> >>> val ssc = new StreamingContext("local[2]" /**TODO change once a cluster >>> is up **/, >>> "AppName", Seconds(1)) >>> >>> >>> I found something that tipped me off that this might work by digging >>> through this mailing list. >>> >>> >>> On Sun, Jul 13, 2014 at 2:00 PM, Walrus theCat <walrusthe...@gmail.com> >>> wrote: >>> >>>> More strange behavior: >>>> >>>> lines.foreachRDD(x => println(x.first)) // works >>>> lines.foreachRDD(x => println((x.count,x.first))) // no output is >>>> printed to driver console >>>> >>>> >>>> >>>> >>>> On Sun, Jul 13, 2014 at 11:47 AM, Walrus theCat <walrusthe...@gmail.com >>>> > wrote: >>>> >>>>> >>>>> Thanks for your interest. >>>>> >>>>> lines.foreachRDD(x => println(x.count)) >>>>> >>>>> And I got 0 every once in a while (which I think is strange, because >>>>> lines.print prints the input I'm giving it over the socket.) >>>>> >>>>> >>>>> When I tried: >>>>> >>>>> lines.map(_->1).reduceByKey(_+_).foreachRDD(x => println(x.count)) >>>>> >>>>> I got no count. >>>>> >>>>> Thanks >>>>> >>>>> >>>>> On Sun, Jul 13, 2014 at 11:34 AM, Tathagata Das < >>>>> tathagata.das1...@gmail.com> wrote: >>>>> >>>>>> Try doing DStream.foreachRDD and then printing the RDD count and >>>>>> further inspecting the RDD. >>>>>> On Jul 13, 2014 1:03 AM, "Walrus theCat" <walrusthe...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have a DStream that works just fine when I say: >>>>>>> >>>>>>> dstream.print >>>>>>> >>>>>>> If I say: >>>>>>> >>>>>>> dstream.map(_,1).print >>>>>>> >>>>>>> that works, too. However, if I do the following: >>>>>>> >>>>>>> dstream.reduce{case(x,y) => x}.print >>>>>>> >>>>>>> I don't get anything on my console. What's going on? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>> >>>>> >>>> >>> >>
