This much isn't related to Spark. Are you looking at /tmp on the executor?
You can use jmap against running JVMs too.
Not sure what error you are facing installing Java
On Thu, Oct 7, 2021, 6:20 PM Kiran Biswal wrote:
> Hello Sean, thanks for the pointer
>
> I followed the guidelines in the lin
This isn't specific to Spark, just use any standard java approach, for
example:
https://dzone.com/articles/how-to-capture-java-heap-dumps-7-options
You need the JDK installed to use jmap
On Mon, Sep 27, 2021 at 1:41 PM Kiran Biswal wrote:
> Thanks Sean.
>
> When executors has only 2gb, execut
Thanks Sean.
When executors has only 2gb, executors restarted every 2/3 hours
with OOMkilled errors
When I increased executir memory to 12 GB and number of cores to 12 (2
executors, 6 cores per executor), the OOMKilled is stopped and restart
happens but the meory usage peaks to 14GB after few hou
It could be 'normal' - executors won't GC unless they need to.
It could be state in your application, if you're storing state.
You'd want to dump the heap to take a first look
On Sat, Sep 25, 2021 at 7:24 AM Kiran Biswal wrote:
> Hello Experts
>
> I have a spark streaming application(DStream). I
