Hi, Barak
It is ok with spark 1.3.0, the problem is with spark 1.4.1.
I don't think spark.storage.memoryFraction will make any sense, because it
is still in heap memory.
------------------ ???????? ------------------
??????: "Barak Gitsis";<[email protected]>;
????????: 2015??8??2??(??????) ????4:11
??????: "Sea"<[email protected]>; "user"<[email protected]>;
????: "rxin"<[email protected]>; "joshrosen"<[email protected]>;
"davies"<[email protected]>;
????: Re: About memory leak in spark 1.4.1
Hi,reducing spark.storage.memoryFraction did the trick for me. Heap doesn't get
filled because it is reserved..
My reasoning is:
I give executor all the memory i can give it, so that makes it a boundary.
From here i try to make the best use of memory I can. storage.memoryFraction is
in a sense user data space. The rest can be used by the system.
If you don't have so much data that you MUST store in memory for performance,
better give spark more space..
ended up setting it to 0.3
All that said, it is on spark 1.3 on cluster
hope that helps
On Sat, Aug 1, 2015 at 5:43 PM Sea <[email protected]> wrote:
Hi, all
I upgrage spark to 1.4.1, many applications failed... I find the heap memory is
not full , but the process of CoarseGrainedExecutorBackend will take more
memory than I expect, and it will increase as time goes on, finally more than
max limited of the server, the worker will die.....
Any can help??
Mode??standalone
spark.executor.memory 50g
25583 xiaoju 20 0 75.5g 55g 28m S 1729.3 88.1 2172:52 java
55g more than 50g I apply
--
-Barak
