The use case I'm working on has a main data stream in which a human needs to
modify what to look for. I'm thinking to implement the main data stream
with Spark Streaming and the things to look for with Spark.
(Better approaches welcome.)
To do this, I have intermixed Spark and Spark Streaming code, and it appears
that the Spark code is not being executed every batch interval. With
details elided, it looks like
val sc = new SparkContext(conf)
val ssc = new StreamingContext(conf, Seconds(10))
ssc.checkpoint(".")
var lines = ssc.textFileStream(dirArg) //
====Spark Streaming code
var linesArray = lines.map( line => (line.split("\t")))
val whiteFd = (new java.io.File(whiteArg)) //
====non-Spark-Streaming code
if (whiteFd.lastModified > System.currentTimeMillis-(timeSliceArg*1000))
{
// read the file into a var
// ====Spark Streaming code
var SvrCum = newState.updateStateByKey[(Int, Time, Time)](updateMyState)
It appears the non-Spark-Streaming code gets executed once at program
initiation but not repeatedly. So, two questions:
1) Is it correct that Spark code does not get executed per batch interval?
2) Is there a definition somewhere of what code will and will not get
executed per batch interval? (I didn't find it in either the Spark or Spark
Streaming programming guides.)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Streaming-which-code-is-not-executed-at-every-batch-interval-tp18071.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
