Re: Create/shutdown objects before/after RDD use (or: Non-serializable classes)

[Tobias Pfeiffer]

Xiangrui,

thanks for your suggestion!

On Sat, May 31, 2014 at 6:12 PM, Xiangrui Meng <men...@gmail.com> wrote:
> One hack you can try is:
>
> rdd.mapPartitions(iter => {
>   val x = new X()
>   iter.map(row => x.doSomethingWith(row)) ++ { x.shutdown(); Iterator.empty }
> })

In fact, I employed a similar hack by now:

rdd.mapPartitions(iter => {
  val x = new X()
  iter.map(row => {
    x.doSomethingWith(row)
    if (!iter.hasNext) x.shutdown()
    row
  })
})

Thanks
Tobias