GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/1080
[SPARK-2144] ExecutorsPage reports incorrect # of RDD blocks
This is reproducible whenever we drop a block because of memory pressure.
When this happens, the StorageStatus should remove the block from its
mapping of BlockId to BlockStatus. Instead, it simply replaces the BlockStatus
with one that has a storage level of `StorageLevel.NONE`. This PR makes sure
that we remove the block from the map if it is no longer cached.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark ui-blocks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1080.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1080
----
commit fcf9f1a381b751cea49e69a266bbf22692be2af0
Author: Andrew Or <[email protected]>
Date: 2014-06-13T22:08:06Z
Remove BlockStatus if it is no longer cached
In StorageStatusListener, a block that is no longer cached has its
BlockStatus updated to reflect this in the StorageStatus blocks map.
What we should really do, however, is to remove this from the map.
Otherwise, the ExecutorsPage still thinks that this is block is cached.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
