Re: commit DB Transaction for each partition

Thu, Aug 27, 2015 at 7:32 PM, Ahmed Nawar wrote: > Dears, > > I needs to commit DB Transaction for each partition,Not for each row. > below didn't work for me. > > > rdd.mapPartitions(partitionOfRecords => { > > DBConnectionInit() &g

Re: Commit DB Transaction for each partition

> >>>> I can not use foreachPartition because i need reuse the new RDD after >>>> some maps. >>>> >>>> >>>> >>>> On Thu, Aug 27, 2015 at 5:11 PM, Cody Koeninger >>>> wrote: >>>> >&

Re: Commit DB Transaction for each partition

oeninger >>> wrote: >>> >>>> >>>> Map is lazy. You need an actual action, or nothing will happen. Use >>>> foreachPartition, or do an empty foreach after the map. >>>> >>>> On Thu, Aug 27, 2015 at 8:53 AM, Ahmed Nawar

Re: Commit DB Transaction for each partition

at 5:11 PM, Cody Koeninger >> wrote: >> >>> >>> Map is lazy. You need an actual action, or nothing will happen. Use >>> foreachPartition, or do an empty foreach after the map. >>> >>> On Thu, Aug 27, 2015 at 8:53 AM, Ahmed Nawar >

Re: Commit DB Transaction for each partition

ing will happen. Use >> foreachPartition, or do an empty foreach after the map. >> >> On Thu, Aug 27, 2015 at 8:53 AM, Ahmed Nawar >> wrote: >> >>> Dears, >>> >>> I needs to commit DB Transaction for each partition,Not for each >>

Commit DB Transaction for each partition

27, 2015 at 5:11 PM, Cody Koeninger wrote: > > Map is lazy. You need an actual action, or nothing will happen. Use > foreachPartition, or do an empty foreach after the map. > > On Thu, Aug 27, 2015 at 8:53 AM, Ahmed Nawar > wrote: > >> Dears, >> >> I nee

commit DB Transaction for each partition

Dears, I needs to commit DB Transaction for each partition,Not for each row. below didn't work for me. rdd.mapPartitions(partitionOfRecords => { DBConnectionInit() val results = partitionOfRecords.map(..) DBConnection.commit() results }) Best regards, Ahmed Atef Naww