Hi,
I have a problem with using accumulators in Spark. As seen on the Spark
website, if you want custom accumulators you can simply extend (with an
object) the AccumulatorParam trait. The problem is that I need to make that
object generic, such as this:
object SeqAccumulatorParam[B] extends AccumulatorParam[Seq[B]] {
override def zero(initialValue: Seq[B]): Seq[B] = Seq[B]()
override def addInPlace(s1: Seq[B], s2: Seq[B]): Seq[B] = s1 ++ s2
}
But this gives me a compile error because objects can't use generic
parameters. My situation doesn't really allow me to define a
SeqAccumulatorParam for each given type since that would lead to a lot of
ugly code duplication.
I have an alternative method, just placing all of the results in an RDD and
then later iterating over them with an accumulator, defined for that single
type, but this would be much nicer.
My question is: is there any other way to create accumulators or some magic
for making generics and objects work?
Cheers,
Johan
