And duh, of course, you can do the setup in that new RDD as well :)
On Wed, Aug 20, 2014 at 1:59 AM, Victor Tso-Guillen wrote:
> How about this:
>
> val prev: RDD[V] = rdd.mapPartitions(partition => { /*setup()*/; partition
> })
> new RDD[V](prev) {
> protected def getPartitions = prev.partit
How about this:
val prev: RDD[V] = rdd.mapPartitions(partition => { /*setup()*/; partition
})
new RDD[V](prev) {
protected def getPartitions = prev.partitions
def compute(split: Partition, context: TaskContext) = {
context.addOnCompleteCallback(() => /*cleanup()*/)
firstParent[V].iter
I think you're looking for foreachPartition(). You've kinda hacked it
out of mapPartitions(). Your case has a simple solution, yes. After
saving to the DB, you know you can close the connection, since you
know the use of the connection has definitely just finished. But it's
not a simpler solution f
> >
> >
> >
> > http://mail-archives.apache.org/mod_mbox/spark-user/201404.mbox/%3CCAF
> > _KkPzpU4qZWzDWUpS5r9bbh=-hwnze2qqg56e25p--1wv...@mail.gmail.com%3E
> >
> >
> >
> > Best regards,
> >
> > Henry
> >
> >
> >
>
@spark.apache.org
Subject: Re: a noob question for how to implement setup and cleanup in Spark map
I think this was a more comprehensive answer recently. Tobias is right that it
is not quite that simple:
http://mail-archives.apache.org/mod_mbox/spark-user/201407.mbox/%3CCAPH-c_O9kQO6yJ4khXUVdO
ards,
>
> Henry
>
>
>
> From: MA33 YTHung1
> Sent: Monday, August 18, 2014 2:42 PM
> To: user@spark.apache.org
> Subject: a noob question for how to implement setup and cleanup in Spark map
>
>
>
> Hi All,
>
>
>
> I’m new to Spark and Scala, just r
gt;
> Henry
>
>
>
> *From:* MA33 YTHung1
> *Sent:* Monday, August 18, 2014 2:42 PM
> *To:* user@spark.apache.org
> *Subject:* a noob question for how to implement setup and cleanup in
> Spark map
>
>
>
> Hi All,
>
>
>
> I’m new to Spark and Scala,
2:42 PM
To: user@spark.apache.org
Subject: a noob question for how to implement setup and cleanup in Spark map
Hi All,
I'm new to Spark and Scala, just recently using this language and love it, but
there is a small coding problem when I want to convert my existing map reduce
code from Ja
Hi All,
I'm new to Spark and Scala, just recently using this language and love it, but
there is a small coding problem when I want to convert my existing map reduce
code from Java to Spark...
In Java, I create a class by extending org.apache.hadoop.mapreduce.Mapper and
override the setup(), ma
