That commit fixed the exact problem you described. That is why I want to confirm that you switched to the master branch. bin/spark-shell doesn't detect code changes, so you need to run ./make-distribution.sh to re-compile Spark first. -Xiangrui
On Tue, Apr 8, 2014 at 9:57 AM, Koert Kuipers <ko...@tresata.com> wrote: > sorry, i meant to say: note that for a cached rdd in the spark shell it > all works fine. but something is going wrong with the SPARK-APPLICATION-UI > in our applications that extensively cache and re-use RDDs > > > On Tue, Apr 8, 2014 at 12:55 PM, Koert Kuipers <ko...@tresata.com> wrote: > >> note that for a cached rdd in the spark shell it all works fine. but >> something is going wrong with the spark-shell in our applications that >> extensively cache and re-use RDDs >> >> >> On Tue, Apr 8, 2014 at 12:33 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> i tried again with latest master, which includes commit below, but ui >>> page still shows nothing on storage tab. >>> koert >>> >>> >>> >>> commit ada310a9d3d5419e101b24d9b41398f609da1ad3 >>> Author: Andrew Or <andrewo...@gmail.com> >>> Date: Mon Mar 31 23:01:14 2014 -0700 >>> >>> [Hot Fix #42] Persisted RDD disappears on storage page if re-used >>> >>> If a previously persisted RDD is re-used, its information disappears >>> from the Storage page. >>> >>> This is because the tasks associated with re-using the RDD do not >>> report the RDD's blocks as updated (which is correct). On stage submit, >>> however, we overwrite any existing >>> >>> Author: Andrew Or <andrewo...@gmail.com> >>> >>> Closes #281 from andrewor14/ui-storage-fix and squashes the >>> following commits: >>> >>> 408585a [Andrew Or] Fix storage UI bug >>> >>> >>> >>> On Mon, Apr 7, 2014 at 4:21 PM, Koert Kuipers <ko...@tresata.com> wrote: >>> >>>> got it thanks >>>> >>>> >>>> On Mon, Apr 7, 2014 at 4:08 PM, Xiangrui Meng <men...@gmail.com> wrote: >>>> >>>>> This is fixed in https://github.com/apache/spark/pull/281. Please try >>>>> again with the latest master. -Xiangrui >>>>> >>>>> On Mon, Apr 7, 2014 at 1:06 PM, Koert Kuipers <ko...@tresata.com> >>>>> wrote: >>>>> > i noticed that for spark 1.0.0-SNAPSHOT which i checked out a few >>>>> days ago >>>>> > (apr 5) that the "application detail ui" no longer shows any RDDs on >>>>> the >>>>> > storage tab, despite the fact that they are definitely cached. >>>>> > >>>>> > i am running spark in standalone mode. >>>>> >>>> >>>> >>> >> >
