Is there a way to pass a custom function to spark to run it on the entire stream? For example, say I have a function which sums up values in each RDD and then across RDDs.
I've tried with map, transform, reduce. They all apply my sum function on 1 RDD. When the next RDD comes the function starts from 0 so the sum of the previous RDD is lost. Does Spark support a way of passing a custom function so that its state is preserved across RDDs and not only within RDD? Thanks -Adrian
