[
https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890629#comment-15890629
]
Thomas Graves commented on SPARK-18769:
---------------------------------------
[~yuming] I already made a comment on that, I don't think we should be looking
at queue configs to do this. We can use the resources returned from the call
to allocate to get a more accurate picture without having to know the internals
of the RM queues. The max capacity of the queue could be the entire cluster
which could be much much larger then a user can actually get based on other
queue configs and who else is using the queue.
[~vanzin] that makes sense but I'm not sure there is anyway around that or
that it matters that much. The yarn api requires 1 container request per
container. It makes sense to try to limit if we don't really need those or
the cluster has no where near those resources. The reply from resource
manager isn't based on the # of requests, it just updates the requests/releases
and returns what the scheduler has allocated to it in between heartbeats.
Scheduler runs async to that that updates the allocations for that application.
ignoring the cluster capacity issue, the hard thing about the dynamic
allocation is determine if the tasks will be quick and thus might not need all
the containers because tasks finish faster then we can allocate executors and
use them.
> Spark to be smarter about what the upper bound is and to restrict number of
> executor when dynamic allocation is enabled
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-18769
> URL: https://issues.apache.org/jira/browse/SPARK-18769
> Project: Spark
> Issue Type: New Feature
> Reporter: Neerja Khattar
>
> Currently when dynamic allocation is enabled max.executor is infinite and
> spark creates so many executor and even exceed the yarn nodemanager memory
> limit and vcores.
> It should have a check to not exceed more that yarn resource limit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
