Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2401#issuecomment-57699365
The Yarn patch sets a default value based on the JVM heap. That works well
for a set of workloads (basically, Java/Scala apps). But the setting itself is
an absolute value; so if you have a pyspark app, or you have a Java/Scala app
that for some reason uses gobs of out-of-heap memory, you just set the absolute
value, you don't need to do funny math.
In your patch the setting itself is a fraction (from the messages so far -
sorry I haven't read the latest version of the code). So if you set the
executor memory to "4g", and your app for some reason uses 30g of out-of-heap
memory, you have to set the multiplier to like 8 or something, instead of just
saying "I need 30g of out-of-heap memory". You're forcing a relationship
between two values that are not necessarily related to each other.
I think the absolute value approach used by Yarn is more user friendly. But
I'm not trying to convince you, I'm just explaining why Yarn does it
differently and why I think it makes better sense.
Note #2485 is not *adding* the setting; the setting already exists. It's
just calculating a sane(r) value for the setting based on other settings. But
the setting still works the same as it did before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
