[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 closed the pull request at: https://github.com/apache/spark/pull/147 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/147#issuecomment-37744167 Continued at #152. Closing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/147#issuecomment-37743968 This should be ready to merge unless other people have more to add. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/147#discussion_r10635968 --- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala --- @@ -136,4 +142,30 @@ class MapOutputTrackerSuite extends FunSuite with

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/147#discussion_r10635967 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -35,13 +35,21 @@ private[spark] case class GetMapOutputStatuses(shuffleId: Int

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-15 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/147#discussion_r10635962 --- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala --- @@ -136,4 +123,47 @@ class MapOutputTrackerSuite extends FunSuite with

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10633506 --- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala --- @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10633389 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala --- @@ -80,187 +81,78 @@ class JobLogger(val user: String, val logDirName: String

[GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...

2014-03-14 Thread andrewor14
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/147 [SPARK-1244] Throw exception if map output status exceeds frame size In the existing code, this fails silently... You can merge this pull request into a Git repository by running: $ git pull

[GitHub] spark pull request: SPARK-1242 Add aggregate to python rdd

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/139#discussion_r10616840 --- Diff: python/pyspark/rdd.py --- @@ -566,8 +566,32 @@ def func(iterator): vals = self.mapPartitions(func).collect() return

[GitHub] spark pull request: SPARK-1242 Add aggregate to python rdd

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/139#discussion_r10616781 --- Diff: python/pyspark/rdd.py --- @@ -566,8 +566,32 @@ def func(iterator): vals = self.mapPartitions(func).collect() return

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616671 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -68,19 +105,53 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616570 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -196,13 +152,46 @@ class DAGScheduler

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616549 --- Diff: core/src/main/scala/org/apache/spark/storage/PutResult.scala --- @@ -20,7 +20,13 @@ package org.apache.spark.storage import

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616539 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerStatusListener.scala --- @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616395 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala --- @@ -22,24 +22,25 @@ import java.text.SimpleDateFormat import java.util

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-14 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10616344 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala --- @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10585234 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -17,28 +17,24 @@ package org.apache.spark.scheduler

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10584092 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10581141 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -895,4 +894,42 @@ private[spark] object Utils extends Logging

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10580856 --- Diff: core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala --- @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10580179 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkReplayerBus.scala --- @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10579167 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -181,15 +178,50 @@ private[spark] class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10578300 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -17,28 +17,24 @@ package org.apache.spark.scheduler

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10576368 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -273,8 +310,9 @@ private[spark] class MapOutputTrackerMaster(conf: SparkConf

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10576181 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -149,14 +151,9 @@ private[spark] class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: SPARK-1236 - Upgrade Jetty to 9.1.3.v20140225.

2014-03-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/113#issuecomment-37555992 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...

2014-03-13 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/89#issuecomment-37554932 @mridulm and @puravaggarwal Thanks for doing this. As a general direction, however, we are actually moving away from MetadataCleaner in the long run - PR#126 is the

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10551095 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10549481 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/126#issuecomment-37484584 This is not specifically related to your patch, but I think we can remove the MetadataCleaner in SparkContext that cleans up persisted RDDs periodically. In particular

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/126#issuecomment-37484105 It's true that finalizers are not predictable, but what this patch is doing is no worse than before. Cleanup() logic is decoupled from finalize(); it can sti

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10543349 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -181,15 +186,49 @@ private[spark] class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10543216 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10543160 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10543007 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10540569 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -181,15 +186,49 @@ private[spark] class MapOutputTracker(conf: SparkConf

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10540432 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10540325 --- Diff: core/src/main/scala/org/apache/spark/ContextCleaner.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1103] [WIP] Automatic garbage collectio...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/126#discussion_r10540258 --- Diff: core/src/main/scala/org/apache/spark/MapOutputTracker.scala --- @@ -50,23 +54,26 @@ private[spark] class MapOutputTrackerMasterActor(tracker

[GitHub] spark pull request: SPARK-1096, a space after comment start style ...

2014-03-12 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/124#discussion_r10534704 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala --- @@ -35,9 +35,9 @@ private[storage] object BlockManagerMessages

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-37372997 @pwendell and @kayousterhout - Thanks for taking the first pass through this patch. I have made the relevant changes we discussed and updated to master (non-trivial as

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504565 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -106,121 +114,154 @@ private[spark] class JobProgressListener(val sc

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504506 --- Diff: core/src/main/scala/org/apache/spark/ui/env/EnvironmentUI.scala --- @@ -19,76 +19,74 @@ package org.apache.spark.ui.env import

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504478 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -43,14 +47,18 @@ case class StorageStatus(blockManagerId: BlockManagerId

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504474 --- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala --- @@ -17,13 +17,17 @@ package org.apache.spark.storage

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504458 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -307,97 +318,93 @@ class BlockManagerMasterActor(val isLocal

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504421 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala --- @@ -40,35 +37,16 @@ private[spark] class SparkListenerBus extends

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504393 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -54,87 +54,53 @@ import org.apache.spark.util.{MetadataCleaner

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10504353 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -229,10 +243,60 @@ object SparkEnv extends Logging { broadcastManager

[GitHub] spark pull request: SPARK-1162 Added top in python.

2014-03-11 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/93#discussion_r10490986 --- Diff: python/pyspark/rdd.py --- @@ -628,6 +656,31 @@ def mergeMaps(m1, m2): m1[k] += v return m1

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10461463 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -30,16 +32,23 @@ import org.apache.spark.scheduler._ * class

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10458614 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterActor.scala --- @@ -50,6 +50,8 @@ class BlockManagerMasterActor(val isLocal

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10458380 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -69,19 +100,55 @@ private[spark] class SparkUI(sc: SparkContext) extends Logging

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10458118 --- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala --- @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10457960 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -30,16 +32,23 @@ import org.apache.spark.scheduler._ * class

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10457836 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -30,16 +32,23 @@ import org.apache.spark.scheduler._ * class

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10455635 --- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala --- @@ -0,0 +1,123 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-10 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-37244892 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-07 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-37054543 I am unable to reproduce the test failure locally. Since the test failure is hidden deep within all the log messages, I have attached them below for convenience

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-07 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36980633 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10368594 --- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala --- @@ -164,9 +167,18 @@ object SparkEnv extends Logging

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10368558 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -19,33 +19,43 @@ package org.apache.spark.scheduler import

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10368525 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala --- @@ -30,16 +32,23 @@ import org.apache.spark.scheduler._ * class

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10367126 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -978,6 +971,11 @@ class DAGScheduler( logDebug("Addit

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10366355 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -978,6 +971,11 @@ class DAGScheduler( logDebug("Addit

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10363030 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -625,6 +653,30 @@ private[spark] class Master(host: String, port: Int

[GitHub] spark pull request: SPARK-1162 Added top in python.

2014-03-06 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/93#discussion_r10362394 --- Diff: python/pyspark/rdd.py --- @@ -628,6 +669,26 @@ def mergeMaps(m1, m2): m1[k] += v return m1

[GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...

2014-03-05 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36821926 Looks like test failed because of git problems. Retesting later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-04 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10284630 --- Diff: core/src/main/scala/org/apache/spark/ui/UIReloader.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-04 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10284454 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-04 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10284428 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-04 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10283514 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io

[GitHub] spark pull request: Patch for SPARK-942

2014-03-04 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10245880 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -71,10 +71,21 @@ private[spark] class CacheManager(blockManager: BlockManager

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10240683 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10239648 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238895 --- Diff: core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala --- @@ -23,9 +23,27 @@ import java.nio.ByteBuffer import

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238901 --- Diff: core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala --- @@ -23,9 +23,27 @@ import java.nio.ByteBuffer import

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238870 --- Diff: core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala --- @@ -23,9 +23,27 @@ import java.nio.ByteBuffer import

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238815 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -71,10 +71,21 @@ private[spark] class CacheManager(blockManager: BlockManager

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10237674 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -847,6 +847,8 @@ class SparkContext( partitions: Seq[Int

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/63#discussion_r10233393 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,6 @@ package org.apache.spark.scheduler * Represents free

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10189124 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -17,21 +17,23 @@ package org.apache.spark.executor

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10187737 --- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10183037 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala --- @@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-27 Thread andrewor14
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/42 [WIP] [SPARK-1132] Persisting Web UI through refactoring the SparkListener interface The fleeting nature of the Spark Web UI has long been a problem reported by many users: The existing Web UI