Xiangrui,
thanks for your suggestion!
On Sat, May 31, 2014 at 6:12 PM, Xiangrui Meng wrote:
> One hack you can try is:
>
> rdd.mapPartitions(iter => {
> val x = new X()
> iter.map(row => x.doSomethingWith(row)) ++ { x.shutdown(); Iterator.empty }
> })
In fact, I employed a similar hack by n
Hi Tobias,
One hack you can try is:
rdd.mapPartitions(iter => {
val x = new X()
iter.map(row => x.doSomethingWith(row)) ++ { x.shutdown(); Iterator.empty }
})
Best,
Xiangrui
On Thu, May 29, 2014 at 11:38 PM, Tobias Pfeiffer wrote:
> Hi,
>
> I want to use an object x in my RDD processing as
Hi,
I want to use an object x in my RDD processing as follows:
val x = new X()
rdd.map(row => x.doSomethingWith(row))
println(rdd.count())
x.shutdown()
Now the problem is that X is non-serializable, so while this works
locally, it does not work in cluster setup. I thought I could do
rdd.mapPart
