Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/4818#issuecomment-76516622
I wouldn't really agree that this is a YARN implementation detail. This is
of course somewhat subjective given that YARN doesn't really document this
behavior, but, speaking with my YARN committer hat on, I'd consider a change
that makes YARN return smaller containers than requested when CPU scheduling is
on a break in compatibility.
Not strongly opposed to adding a config, but, if we do, I think it's better
for the default to optimize for ease of use over the remote possibility that a
future version of YARN might change behavior in this way.
Also, whatever behavior we decide on, it would be good (and should be
straightforward) to add a test for it in `YarnAllocatorSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
