On Thu, Jan 5, 2017 at 4:58 PM, Kay Ousterhout <k...@eecs.berkeley.edu> wrote: > But is there any non-memory-leak reason why the tests should need more > memory? In theory each test should be cleaning up it's own Spark Context > etc. right? My memory is that OOM issues in the tests in the past have been > indicative of memory leaks somewhere.
It can always be some memory leak; if we increase the memory settings and OOMs still happen, that would be a good indication. Also if the same tests tend to fail (even if unreliably). Normally some existing code / feature requiring more memory than before, especially some non-trivial amount, is suspicious. But sometimes new features / new tests require more memory than configured in the build scripts, and sometimes blow up. -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
