[
https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891610#comment-15891610
]
Xuefu Zhang commented on SPARK-18769:
-------------------------------------
Just as fyi, the problem is real and happens when allocation attempts are made
as long as there are pending tasks (which is in line with dynamic allocation).
However, it's pointless when all containers are already taken and further
attempts are very likely in vain, which adversely creates pressure on event
processing in Spark driver and may also have impact on YARN RM.
I don't know what's the best solution for this, Maybe Spark can just try to
allocate all it needs upfront and update (tune down) the allocation request as
the job progresses when necessary.
As a workaround, we have to set an artificial upper limit (something like
2500), which helps a lot.
> Spark to be smarter about what the upper bound is and to restrict number of
> executor when dynamic allocation is enabled
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-18769
> URL: https://issues.apache.org/jira/browse/SPARK-18769
> Project: Spark
> Issue Type: New Feature
> Reporter: Neerja Khattar
>
> Currently when dynamic allocation is enabled max.executor is infinite and
> spark creates so many executor and even exceed the yarn nodemanager memory
> limit and vcores.
> It should have a check to not exceed more that yarn resource limit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
