Hi,
I've got a socketTextStream through which I'm reading input. I have three
Dstreams, all of which are the same window operation over that
socketTextStream. I have a four core machine. As we've been covering
lately, I have to give a "cores" parameter to my StreamingSparkContext:
ssc = new StreamingContext("local[4]" /**TODO change once a cluster is up
**/,
"AppName", Seconds(1))
Now, I have three dstreams, and all I ask them to do is print or count. I
should preface this with the statement that they all work on their own.
dstream1 // 1 second window
dstream2 // 2 second window
dstream3 // 5 minute window
If I construct the ssc with "local[8]", and put these statements in this
order, I get prints on the first one, and zero counts on the second one:
ssc(local[8]) // hyperthread dat sheezy
dstream1.print // works
dstream2.count.print // always prints 0
If I do this, this happens:
ssc(local[4])
dstream1.print // doesn't work, just gives me the Time: .... ms message
dstream2.count.print // doesn't work, prints 0
ssc(local[6])
dstream1.print // doesn't work, just gives me the Time: .... ms message
dstream2.count.print // works, prints 1
Sometimes these results switch up, seemingly at random. How can I get
things to the point where I can develop and test my application locally?
Thanks
