Hi Spark ! I found out why my RDD's werent coming through in my spark
stream.
It turns out you need the onStart() needs to return , it seems - i.e. you
need to launch the worker part of your
start process in a thread. For example....
def onStartMock():Unit ={
val future = new Thread(new Runnable() {
def run() {
for(x <- 1 until 1000000000) {
val newMem = Runtime.getRuntime.freeMemory()/12188091;
if(newMem != lastMem){
System.out.println(" in thread : " + newMem);
}
lastMem=newMem;
store(mockStatus);
}
}});
Hope that helps somebody in the same situation. FYI Its in the docs :)
* {{{
* class MyReceiver(storageLevel: StorageLevel) extends
NetworkReceiver[String](storageLevel) {
* def onStart() {
* // Setup stuff (start threads, open sockets, etc.) to start
receiving data.
* // Must start new thread to receive data, as onStart() must be
non-blocking.
*
* // Call store(...) in those threads to store received data into
Spark's memory.
*
* // Call stop(...), restart(...) or reportError(...) on any
thread based on how
* // different errors needs to be handled.
*
* // See corresponding method documentation for more details
* }
*
* def onStop() {
* // Cleanup stuff (stop threads, close sockets, etc.) to stop
receiving data.
* }
* }
* }}}
