Re: Executor Lost with StorageLevel.MEMORY_AND_DISK_SER

2015-02-10 Thread Marius Soutier
, > Vinay > > -Original Message- > From: Marius Soutier [mailto:mps@gmail.com] > Sent: Monday, February 09, 2015 2:19 AM > To: user > Subject: Executor Lost with StorageLevel.MEMORY_AND_DISK_SER > > Hi there, > > I'm trying to improve performance on a job that

Executor Lost with StorageLevel.MEMORY_AND_DISK_SER

2015-02-09 Thread Marius Soutier
Hi there, I’m trying to improve performance on a job that has GC troubles and takes longer to compute simply because it has to recompute failed tasks. After deferring object creation as much as possible, I’m now trying to improve memory usage with StorageLevel.MEMORY_AND_DISK_SER and a custom K