[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36419075 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36418482 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36418483 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12942/ --- If your project i

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36417618 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36417620 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-28 Thread Mridul Muralidharan
On Sat, Mar 1, 2014 at 2:05 AM, Patrick Wendell wrote: > Hey, > > Thanks everyone for chiming in on this. I wanted to summarize these > issues a bit particularly wrt the constituents involved - does this > seem accurate? > > = Spark Users = > In general those linking against Spark should be totall

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10189405 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager:

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36416731 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36416732 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12941/ --- If your pr

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36416706 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36416705 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10189124 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -17,21 +17,23 @@ package org.apache.spark.executor +imp

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414847 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414848 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12940/ --- If your pr

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-28 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/11#issuecomment-36414832 @pwendell , I just updated the code In the latest update, I make the checking only applicable to FileOutputFormat, the difference with your suggestion is that I a

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414656 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414655 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414645 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414646 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12939/ --- If your pr

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10188872 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager:

[GitHub] spark pull request: SPARK-1151: Update dev merge script to use spa...

2014-02-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/47 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: [SPARK-972] Added detailed callsite info for V...

2014-02-28 Thread jyotiska
Github user jyotiska commented on the pull request: https://github.com/apache/spark/pull/34#issuecomment-36414522 Can one of the admins look into this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10188851 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager:

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36414270 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36414271 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12938/ --- If your project i

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414262 Your suggestion of @tgravescs tying this yarn-alpha I think is strictly better. Just updated the patch. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414245 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36414246 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36414229 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36414230 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12937/ --- If your pr

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413838 It is running now. Let's wait for Jenkins to come back. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413688 It seems that there are still some problems. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413637 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12936/ --- If your pr

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413636 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413558 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36413557 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36413556 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413559 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1151: Update dev merge script to use spa...

2014-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/47#issuecomment-36413458 Thanks tom - merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413420 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413377 It's not you. There was somehow two Jenkins pull request builder setup ... I Just removed one of them. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/43#issuecomment-36413341 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413338 @rxin It seems that I rebased and squashed the commits which made Jenkins unhappy. Could you ask Jenkins to build again? Thanks. --- If your project is set up for it, you c

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-28 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/11#discussion_r10188629 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -712,6 +713,10 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413276 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413273 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413274 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413277 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder%20(incubator)/1/ ---

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/40#discussion_r10188617 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala --- @@ -149,7 +149,14 @@ object GradientDescent extends Logging {

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413264 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413254 Jenkins, add to whitelist. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36413266 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/11#discussion_r10188570 --- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala --- @@ -712,6 +713,10 @@ class PairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread rxin
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/40#discussion_r10188555 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala --- @@ -149,7 +149,14 @@ object GradientDescent extends Logging {

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36410331 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36410332 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12935/ --- If your project i

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10187737 --- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36408974 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36408973 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36408986 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Patch for SPARK-942

2014-02-28 Thread kellrott
GitHub user kellrott opened a pull request: https://github.com/apache/spark/pull/50 Patch for SPARK-942 This is a port of a pull request original targeted at incubator-spark: https://github.com/apache/incubator-spark/pull/180 Essentially if a user returns a generative itera

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10186927 --- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10186902 --- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10186874 --- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10186680 --- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10186518 --- Diff: core/src/main/scala/org/apache/spark/ui/UISparkListener.scala --- @@ -0,0 +1,210 @@ +/* + * Licensed to the Apache Software Foundation (ASF) u

[GitHub] spark pull request: [SPARK-1100] prevent Spark from overwriting di...

2014-02-28 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/11#issuecomment-36406325 I changed the code and tested in local side, mind reviewing it again? @pwendell --- If your project is set up for it, you can reply to this email and have your reply ap

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-28 Thread Mark Hamstra
Couple of comments: 1) Whether the Spark POM is produced by SBT or Maven shouldn't matter for those who just need to link against published artifacts, but right now SBT and Maven do not produce equivalent POMs for Spark -- I think 2) Incremental builds using Maven are trivially more difficult

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10184513 --- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala --- @@ -17,21 +17,23 @@ package org.apache.spark.executor +impor

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36400692 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36400693 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12934/ --- If your project i

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10183713 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -194,12 +194,19 @@ class DAGScheduler( } }

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10183527 --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala --- @@ -415,6 +415,7 @@ class JavaSparkContext(val sc: SparkContext) extends Ja

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36398305 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36398886 I'll give it a try. Any reason we don't just tie this to the yarn-alpha profile? Or does it not apply to the hadoop 2.0.2 type builds? --- If your project is set up for

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10183222 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala --- @@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String)

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10183037 --- Diff: core/src/main/scala/org/apache/spark/scheduler/JobLogger.scala --- @@ -339,8 +217,8 @@ class JobLogger(val user: String, val logDirName: String)

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36398321 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36398304 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread dbtsai
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/40#discussion_r10181447 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala --- @@ -149,7 +149,13 @@ object GradientDescent extends Logging {

New blog post on Spark + Parquet + Scrooge

2014-02-28 Thread Evan Chan
http://engineering.ooyala.com/blog/using-parquet-and-scrooge-spark Enjoy! By the way, I was not able to subscribe to the user-digest list for some reason. The help email claims Similar addresses exist for the digest list: But if you send a email to the digest-subscribe it bounces ba

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36392299 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36392301 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12933/ --- If your project i

Re: [DISCUSS] Necessity of Maven *and* SBT Build in Spark

2014-02-28 Thread Patrick Wendell
Hey, Thanks everyone for chiming in on this. I wanted to summarize these issues a bit particularly wrt the constituents involved - does this seem accurate? = Spark Users = In general those linking against Spark should be totally unaffected by the build choice. Spark will continue to publish well-

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-02-28 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10180027 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager:

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36389809 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36389827 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36389807 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1121: Use 'avro' profile in Maven.

2014-02-28 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/49 SPARK-1121: Use 'avro' profile in Maven. This lets us explicitly include Avro based on a profile for 0.23.X builds. It makes me sad how convoluted it is to express this logic in Maven. @tgrave

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/40#discussion_r10178907 --- Diff: mllib/src/test/scala/org/apache/spark/mllib/optimization/GradientDescentSuite.scala --- @@ -104,4 +104,45 @@ class GradientDescentSuite extends FunSui

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-02-28 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/40#discussion_r10178847 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala --- @@ -149,7 +149,13 @@ object GradientDescent extends Logging {

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-28 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/38#discussion_r10178681 --- Diff: bin/pyspark --- @@ -58,7 +58,8 @@ if [ -n "$IPYTHON_OPTS" ]; then IPYTHON=1 fi -if [[ "$IPYTHON" = "1" ]] ; then +# Only us

[GitHub] spark pull request: SPARK-1134 pyspark only uses ipython if starte...

2014-02-28 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/38#issuecomment-36387844 I'd like to hold off on merging this until we see how https://github.com/ipython/ipython/pull/5226 is resolved. It looks like IPython executes `PYTHONSTARTUP` for non-in

New JIRA ticket: cleaning up app-* folders

2014-02-28 Thread Evan Chan
Hey guys, FYI, I just filed a new ticket, https://spark-project.atlassian.net/browse/SPARK-1154, concerning that Spark leaves tons of app-* folders on disk, which can quickly fill up the disk. Feel free to comment. I believe it is better for Spark to clean itself up; ie users should not need a cr

Re: Spark JIRA

2014-02-28 Thread Evan Chan
Ok, will continue to work with the existing one for now. On Fri, Feb 28, 2014 at 11:36 AM, Josh Rosen wrote: > Apache INFRA has been trying to import our JIRA issues since June 2013: > https://issues.apache.org/jira/browse/INFRA-6419. It seems that JIRA > introduces minor import incompatibilitie

Re: Spark JIRA

2014-02-28 Thread Josh Rosen
Apache INFRA has been trying to import our JIRA issues since June 2013: https://issues.apache.org/jira/browse/INFRA-6419. It seems that JIRA introduces minor import incompatibilities with every update to JIRA OnDemand, so it's been a challenge to get the Apache and Atlasssian JIRA versions to sync

Re: Spark JIRA

2014-02-28 Thread Nan Zhu
I think they are working on it? https://issues.apache.org/jira/browse/SPARK Best, -- Nan Zhu On Friday, February 28, 2014 at 2:29 PM, Evan Chan wrote: > Hey guys, > > There is no plan to move the Spark JIRA from the current > https://spark-project.atlassian.net/ > > right? > > -- > -- >

Spark JIRA

2014-02-28 Thread Evan Chan
Hey guys, There is no plan to move the Spark JIRA from the current https://spark-project.atlassian.net/ right? -- -- Evan Chan Staff Engineer e...@ooyala.com |

[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...

2014-02-28 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/29 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-02-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36383623 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12932/ --- If your pr

  1   2   >