Re: Spark Streaming 1.6 accumulating state across batches for joins

Please disregard the "window" functions...it turns out that was development code. Everything else is correct. val rawLEFT: DStream[String] = ssc.textFileStream(dirLEFT). window(Seconds(30)) val rawRIGHT: DStream[String] = ssc.textFileStream(dirRIGHT). window(Seconds(30)) should be val rawLEFT

Spark Streaming 1.6 accumulating state across batches for joins

Hello folks, I'm on the newest spark 1.6.0-SNAPSHOT Spark Streaming with the new trackStateByKey API. I'm trying to do something fairly simple that requires knowing state across minibatches, so I am trying to see if it can be done. I basically have two types of data to do a join, left-side and ri