What problem are you having? you will have to trigger an action at the end
to execute this piece of code. Like:
rdd.mapPartitions(partitionOfRecords => {
DBConnectionInit()
val results = partitionOfRecords.map(..)
DBConnection.commit()
results
})*.count()*
Thanks
Best Regards
On Thu,
Thanks a lot for your support. It is working now.
I wrote it like below
val newRDD = rdd.mapPartitions { partition => {
val result = partition.map(.)
result
}
}
newRDD.foreach {
}
On Thu, Aug 27, 2015 at 10:34 PM, Cody Koeninger wrote:
> This job contains a spark output action, an
This job contains a spark output action, and is what I originally meant:
rdd.mapPartitions {
result
}.foreach {
}
This job is just a transformation, and won't do anything unless you have
another output action. Not to mention, it will exhaust the iterator, as
you noticed:
rdd.mapPartitions {
Yes, of course, I am doing that. But once i added results.foreach(row=> {})
i pot empty RDD.
rdd.mapPartitions(partitionOfRecords => {
DBConnectionInit()
val results = partitionOfRecords.map(..)
DBConnection.commit()
results.foreach(row=> {})
results
})
On Thu, Aug 27, 2015 at 10:
You need to return an iterator from the closure you provide to mapPartitions
On Thu, Aug 27, 2015 at 1:42 PM, Ahmed Nawar wrote:
> Thanks for foreach idea. But once i used it i got empty rdd. I think
> because "results" is an iterator.
>
> Yes i know "Map is lazy" but i expected there is solutio
