Re: Memory-efficient successive calls to repartition()

data.unpersist() # unpersist previous version data=data2 Help and suggestions on this would be greatly appreciated! Thanks a lot! -- View this message in contex

Re: Memory-efficient successive calls to repartition()

ions on this would be greatly appreciated! Thanks a lot! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Memory-efficient-su

Re: Memory-efficient successive calls to repartition()

. This >>> is an >>> expensive >>> operation but I only need to do it every now and then. >>> However it >>> seems that >>> I am doing something wrong because as the iterations go

Re: Memory-efficient successive calls to repartition()

on't >> get if I don't >> repartition. >> >> Here's a basic piece of code which reproduces the problem: >> >> data = >> sc.textFile("ImageNet_gist_train.txt",50).map(parseLine).cac

Re: Memory-efficient successive calls to repartition()

Thanks a lot! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Memory-efficient-successive-calls-to-repartition-tp24358.html Sent from the Apache Spark User List mailing list archive at Nabble.c

Re: Memory-efficient successive calls to repartition()

g? I tried the following but it doesn't solve >> the issue: >> >> for i in range(1000): >> data2=data.repartition(50).persist() >> data2.count() # materialize rdd >> data.unpersist() # unpersist previous versi

Re: Memory-efficient successive calls to repartition()

() # materialize rdd data.unpersist() # unpersist previous version data=data2 Help and suggestions on this would be greatly appreciated! Thanks a lot! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Memory-efficien

Re: Memory-efficient successive calls to repartition()

unpersist() # unpersist previous version > data=data2 > > > Help and suggestions on this would be greatly appreciated! Thanks a lot! > > > > > -- > View this message in context: > http://apache-s

Re: Memory-efficient successive calls to repartition()

unpersist() # unpersist previous version > data=data2 > > > Help and suggestions on this would be greatly appreciated! Thanks a lot! > > > > > -- > View this message in context: > http://apache-s

Memory-efficient successive calls to repartition()

data2=data.repartition(50).persist() data2.count() # materialize rdd data.unpersist() # unpersist previous version data=data2 Help and suggestions on this would be greatly appreciated! Thanks a lot! -- View this message in context: http://apache-spark-user-list.1001560.