Re: Ever increasing physical memory for a Spark Application in YARN

2016-05-03 Thread Nitin Goyal
Hi Daniel, I could indeed discover the problem in my case and it turned out to be a bug at parquet side and I had raised and contributed to the following issue :- https://issues.apache.org/jira/browse/PARQUET-353 Hope this helps! Thanks -Nitin On Mon, May 2, 2016 at 9:15 PM, Daniel Darabos <

Re: Ever increasing physical memory for a Spark Application in YARN

2016-05-02 Thread Daniel Darabos
Hi Nitin, Sorry for waking up this ancient thread. That's a fantastic set of JVM flags! We just hit the same problem, but we haven't even discovered all those flags for limiting memory growth. I wanted to ask if you ever discovered anything further? I see you also set -XX:NewRatio=3. This is a ver