Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/1165#issuecomment-48415357
I ran a few perf tests on 5 r3.8xlarge nodes, caching with both
`MEMORY_ONLY` and `MEMORY_AND_DISK`, and observed no significant performance
regression.
In terms of correctness, you can run the following in spark-shell as a
sanity check
```
sc.parallelize(1 to 20 * 1000 * 1000, 1).persist().count()
```
With the default configuration (512MB executor memory), this job OOMs
before the changes in this patch, but no longer does after the changes (because
it doesn't attempt to cache the block in memory).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
