Re: lost executor due to large shuffle spill memory

2016-04-06 Thread Michael Slavitch
> > both_reversed = user_reversed.join(product_reverse).map(lambda r: > > (r[1][0][0], r[1][1], r[1][0][1])) > > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > > x[2])).saveAsTextFile(recommendation_path) > > > > Both user and products are

Re: lost executor due to large shuffle spill memory

2016-04-06 Thread Lishu Liu
n(user_reverse).map(lambda r: (r[1][0][0], (r[1][1], > > r[1][0][1]))) > > both_reversed = user_reversed.join(product_reverse).map(lambda r: > > (r[1][0][0], r[1][1], r[1][0][1])) > > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > > x[2])).s

Re: lost executor due to large shuffle spill memory

2016-04-05 Thread Michael Slavitch
> (r[1][0][0], r[1][1], r[1][0][1])) > both_reversed.map(lambda x: '{}|{}|{}'.format(x[0], x[1], > x[2])).saveAsTextFile(recommendation_path) > > Both user and products are (uuid, index) tuples. > > > > -- > View this message in context: > http://apac

lost executor due to large shuffle spill memory

2016-04-05 Thread lllll
x[1], x[2])).saveAsTextFile(recommendation_path) Both user and products are (uuid, index) tuples. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/lost-executor-due-to-large-shuffle-spill-memory-tp26683.html Sent from the Apache Spa