You can do the following.
var globalMax = ...
dstreamOfNumericalType.foreachRDD( rdd => {
globalMax = math.max(rdd.max, globalMax)
})
globalMax will keep getting updated after every batch
TD
On Thu, Aug 7, 2014 at 5:31 PM, bumble123 <tc1...@att.com> wrote:
> I can't figure out how to use Spark Streaming to find the max of a 5 second
> batch of data and keep updating the max every 5 seconds. How would I do
> this?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkStreaming-to-find-the-max-of-a-dataset-tp11734.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
