Re: Spark streaming spilling all the data to disk even if memory available

; type of consumer you are using etc? > > Thanks > Best Regards > > On Tue, Mar 29, 2016 at 6:10 PM, Mayur Mohite > wrote: > >> Hi, >> >> We are running spark streaming app on a single machine and we have >> configured spark executor memory to 30G. &g

Spark streaming spilling all the data to disk even if memory available

Hi, We are running spark streaming app on a single machine and we have configured spark executor memory to 30G. We noticed that after running the app for 12 hours, spark streaming started spilling ALL the data to disk even though we have configured sufficient memory for spark to use for storage.