Re: Spark off heap memory leak on Yarn with Kafka direct stream

-Jul-2015, at 12:43 am, Cody Koeninger wrote: > > Does the issue only happen when you have no traffic on the topic? > > Have you profiled to see what's using heap space? > > > On Mon, Jul 13, 2015 at 1:05 PM, Apoorva Sareen <mailto:apoorva.sar...@gmail.com>>

Spark off heap memory leak on Yarn with Kafka direct stream

Hi, I am running spark streaming 1.4.0 on Yarn (Apache distribution 2.6.0) with java 1.8.0_45 and also Kafka direct stream. I am also using spark with scala 2.11 support. The issue I am seeing is that both driver and executor containers are gradually increasing the physical memory usage till a