setup and cleanup function in spark

To implement setup/cleanup function in Spark we follow the pattern below as discussed here . rdd.mapPartitions { partition => if (!partition.isEmpty) { // Some setup code here part

Re: Is there setup and cleanup function in spark?

I see. Agree that lazy eval is not suitable for proper setup and teardown. We also abandoned it due to inherent incompatibility between implicit and lazy. It was fun to come up this trick though. Jianshi On Tue, Nov 18, 2014 at 10:28 AM, Tobias Pfeiffer wrote: > Hi, > > On Fri, Nov 14, 2014 at

Re: Is there setup and cleanup function in spark?

Hi, On Fri, Nov 14, 2014 at 2:49 PM, Jianshi Huang wrote: > Ok, then we need another trick. > > let's have an *implicit lazy var connection/context* around our code. And > setup() will trigger the eval and initialization. > Due to lazy evaluation, I think having setup/teardown is a bit tricky.

Re: Is there setup and cleanup function in spark?

Ok, then we need another trick. let's have an *implicit lazy var connection/context* around our code. And setup() will trigger the eval and initialization. The implicit lazy val/var trick is actually invented by Kevin. :) Jianshi On Fri, Nov 14, 2014 at 1:41 PM, Cheng Lian wrote: > If you’re

Re: Is there setup and cleanup function in spark?

If you’re just relying on the side effect of |setup()| and |cleanup()| then I think this trick is OK and pretty cleaner. But if |setup()| returns, say, a DB connection, then the |map(...)| part and |cleanup()| can’t get the connection object. On 11/14/14 1:20 PM, Jianshi Huang wrote: So can

Re: Is there setup and cleanup function in spark?

So can I write it like this? rdd.mapPartition(i => setup(); i).map(...).mapPartition(i => cleanup(); i) So I don't need to mess up the logic and still can use map, filter and other transformations for RDD. Jianshi On Fri, Nov 14, 2014 at 12:20 PM, Cheng Lian wrote: > If you’re looking for ex

Re: Is there setup and cleanup function in spark?

If you’re looking for executor side setup and cleanup functions, there ain’t any yet, but you can achieve the same semantics via |RDD.mapPartitions|. Please check the “setup() and cleanup” section of this blog from Cloudera for details: http://blog.cloudera.com/blog/2014/09/how-to-translate-f

Re: Is there setup and cleanup function in spark?

Where do you want the setup and cleanup functions to run? Driver or the worker nodes? Jianshi On Fri, Nov 14, 2014 at 10:44 AM, Dai, Kevin wrote: > HI, all > > > > Is there setup and cleanup function as in hadoop mapreduce in spark which > does some initialization and cleanup work? > > > > Bes

Is there setup and cleanup function in spark?

HI, all Is there setup and cleanup function as in hadoop mapreduce in spark which does some initialization and cleanup work? Best Regards, Kevin.