RE: Memory problems and missing heartbeats

gt; first problem. > > Ximo. > > -Mensaje original- > De: Arkadiusz Bicz [mailto:arkadiusz.b...@gmail.com] > Enviado el: martes, 16 de febrero de 2016 16:06 > Para: Iulian Dragoș > CC: JOAQUIN GUANTER GONZALBEZ ; > user@spark.apache.org > Asunto: Re: Memory problems and missi

RE: Memory problems and missing heartbeats

user@spark.apache.org Asunto: Re: Memory problems and missing heartbeats I had similar as #2 problem when I used lot of caching and then doing shuffling It looks like when I cached too much there was no enough space for other spark tasks and it just hang on. That you can try to cache less and see if improve,

RE: Memory problems and missing heartbeats

A GC pause fits nicely with what I’m seeing. Many thanks for the link! Ximo De: Iulian Dragoș [mailto:iulian.dra...@typesafe.com] Enviado el: martes, 16 de febrero de 2016 15:14 Para: JOAQUIN GUANTER GONZALBEZ CC: user@spark.apache.org Asunto: Re: Memory problems and missing heartbeats

Re: Memory problems and missing heartbeats

I had similar as #2 problem when I used lot of caching and then doing shuffling It looks like when I cached too much there was no enough space for other spark tasks and it just hang on. That you can try to cache less and see if improve, also executor logs help a lot (watch out logs with informatio

Re: Memory problems and missing heartbeats

Regarding your 2nd problem, my best guess is that you’re seeing GC pauses. It’s not unusual, given you’re using 40GB heaps. See for instance this blog post >From conducting numerous tests, we have concluded that unless

RE: Memory problems and missing heartbeats

Bumping this thread in hopes that someone will answer. Ximo -Mensaje original- De: JOAQUIN GUANTER GONZALBEZ [mailto:joaquin.guantergonzal...@telefonica.com] Enviado el: lunes, 15 de febrero de 2016 16:43 Para: user@spark.apache.org Asunto: Memory problems and missing heartbeats Hello,