[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36489261 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/42#issuecomment-36489262 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12963/ --- If your project i

[GitHub] spark pull request: SPARK-1173. (#2) Fix typo in Java streaming ex...

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/65 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: SPARK-1173. Improve scala streaming docs.

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/64 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: update proportion of memory

2014-03-03 Thread CrazyJvm
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/66 update proportion of memory The default value of "spark.storage.memoryFraction" has been change from 0.66 to 0.6 . So it should be 60% of the memory to cache while 40% used for task execution. You

[GitHub] spark pull request: MLI-2: Start adding k-fold cross validation to...

2014-03-03 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/18#discussion_r10203849 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala --- @@ -62,6 +67,20 @@ object MLUtils { } /** + * Return a k e

[GitHub] spark pull request: update proportion of memory

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/66#issuecomment-36493034 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread jyotiska
GitHub user jyotiska opened a pull request: https://github.com/apache/spark/pull/67 Fixed API docs link in Python programming guide You can merge this pull request into a Git repository by running: $ git pull https://github.com/jyotiska/spark pyspark_docs3 Alternatively you c

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497237 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/67#issuecomment-36497225 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/67#issuecomment-36497227 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497236 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497343 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497344 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497569 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36497570 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread jyotiska
GitHub user jyotiska opened a pull request: https://github.com/apache/spark/pull/68 Updated the formatting of code blocks using Github flavored markdown Github does not work with "highlight" to format code blocks which turns them into plain text. In this PR, I used Github flavored m

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501325 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501327 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12968/ --- If your project i

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/67#issuecomment-36501329 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12965/ --- If your project i

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501338 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12966/ --- If your project i

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/67#issuecomment-36501328 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501335 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501336 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12967/ --- If your project i

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501334 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/68#issuecomment-36501475 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501482 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36501481 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/68#issuecomment-36501476 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/68#issuecomment-36505124 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12969/ --- If your project i

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36505127 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12970/ --- If your project i

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/68#issuecomment-36505123 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36505126 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

Re: [GitHub] spark pull request: MLI-2: Start adding k-fold cross validation to...

2014-03-03 Thread Mark Hamstra
'to' is an exception to the usual rule, so (1 to folds).map {... } would be the best form. On Mon, Mar 3, 2014 at 1:02 AM, holdenk wrote: > Github user holdenk commented on a diff in the pull request: > > https://github.com/apache/spark/pull/18#discussion_r10203849 > > --- Diff: > mllib

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/63#issuecomment-36519884 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/63#issuecomment-36519886 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[SUGGESTION] suggest contributors to run sbt scalastyle before run sbt test

2014-03-03 Thread Nan Zhu
Hi, all I noticed this because…my two PRs failed for the style error (exceeding for 3 - 5 chars) yesterday Maybe we can explicitly suggest contributors to run sbt scalastyle before they run test cases https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark Just add one sente

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10215150 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10215322 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10216010 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -135,6 +135,8 @@ class SparkContext( val isLocal = (master == "local" ||

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/63#issuecomment-36526800 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12971/ --- If your project i

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/63#issuecomment-36526798 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10216556 --- Diff: core/src/main/scala/org/apache/spark/network/Connection.scala --- @@ -18,25 +18,27 @@ package org.apache.spark.network import org.apa

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10216724 --- Diff: core/src/main/scala/org/apache/spark/network/ConnectionManager.scala --- @@ -557,7 +754,54 @@ private[spark] class ConnectionManager(port: Int, con

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10219868 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10220939 --- Diff: core/src/main/scala/org/apache/spark/network/SecurityMessage.scala --- @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10221207 --- Diff: core/src/main/scala/org/apache/spark/network/Connection.scala --- @@ -18,25 +18,27 @@ package org.apache.spark.network import org.apa

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r10221836 --- Diff: docs/configuration.md --- @@ -477,6 +505,21 @@ Apart from these, the following properties are also available, and may be useful Whe

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36542188 Thanks for the detailed review Patrick. I've updated based on the comments except for renaming the Handlers to Servlet and changing to use SparkConf. I will make those

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r1063 --- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala --- @@ -0,0 +1,259 @@ +/* + * Licensed to the Apache Software Foundation (ASF) und

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/33#discussion_r1084 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -41,56 +46,103 @@ private[spark] object JettyUtils extends Logging { type Resp

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36547975 Build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36547977 Build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36548137 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12972/ --- If your pr

[GitHub] spark pull request: Add Security to Spark - Akka, Http, Connection...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/33#issuecomment-36548135 Build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10225217 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskMan

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10225873 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10226896 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10226983 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread ScrapCodes
Github user ScrapCodes commented on a diff in the pull request: https://github.com/apache/spark/pull/17#discussion_r10227199 --- Diff: extras/java8-tests/README.md --- @@ -0,0 +1,15 @@ +# Java 8 test suites. + +These tests are bundled with spark and run if you have java

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10227330 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala --- @@ -146,6 +146,12 @@ object BlockFetcherIterator { }

[GitHub] spark pull request: Fixed API docs link in Python programming guid...

2014-03-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/67#issuecomment-36562029 Hi @jyotiska These docs are not meant to be consumed directly as markdown files. They are meant to be generated using jekyll (run jekyll build in docs folder), and t

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/63#discussion_r10230140 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,6 @@ package org.apache.spark.scheduler * Represents fre

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/63#discussion_r10230444 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,6 @@ package org.apache.spark.scheduler * Represents fre

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread markhamstra
Github user markhamstra commented on a diff in the pull request: https://github.com/apache/spark/pull/63#discussion_r10232174 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,6 @@ package org.apache.spark.scheduler * Represents free

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-03 Thread DB Tsai
Hi Deb, 1) The LBFGS code I published to maven central is here, https://github.com/dbtsai/lbfgs You can fork it, and publish it locally so that mllib can use it. I also want to add orthant wise for L1, so we can work together to make it happen. I think ideally, we can rewrite it in scala, and mo

[GitHub] spark pull request: Removed accidentally checked in comment

2014-03-03 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/61#issuecomment-36569794 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request: Removed accidentally checked in comment

2014-03-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/61#issuecomment-36569896 I merged this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark pull request: update proportion of memory

2014-03-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/66#issuecomment-36570035 Thanks. I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: [Proposal] SPARK-1171: simplify the implementa...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/63#discussion_r10233393 --- Diff: core/src/main/scala/org/apache/spark/scheduler/WorkerOffer.scala --- @@ -21,4 +21,6 @@ package org.apache.spark.scheduler * Represents free r

Re: [SUGGESTION] suggest contributors to run sbt scalastyle before run sbt test

2014-03-03 Thread Reynold Xin
Thanks for the suggestion. Just did it. On Mon, Mar 3, 2014 at 7:38 AM, Nan Zhu wrote: > Hi, all > > I noticed this because...my two PRs failed for the style error (exceeding > for 3 - 5 chars) yesterday > > Maybe we can explicitly suggest contributors to run sbt scalastyle before > they run te

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/68#issuecomment-36570874 This breaks Jekyll's code block formatting: ![image](https://f.cloud.github.com/assets/50748/2315605/c80bfdd0-a325-11e3-9cf6-8051590b96bc.png) As @rxin s

[GitHub] spark pull request: Removed accidentally checked in comment

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/61 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: update proportion of memory

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/66 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Updated the formatting of code blocks using Gi...

2014-03-03 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/68#discussion_r10235312 --- Diff: docs/python-programming-guide.md --- @@ -6,7 +6,7 @@ title: Python Programming Guide The Spark Python API (PySpark) exposes the Spark program

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36574208 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36574207 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Remove the remoteFetchTime metric.

2014-03-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/62#issuecomment-36575370 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-03 Thread Debasish Das
Hi DB, I have forked the LBFGS repo. The features that I want in it are the following: a. OWL-QN for solving L1 natively in BFGS b. Bound constraints in BFGS : I saw you have converted the fortran code. Is there a license issue ? I can help in getting that up to speed as well. c. Few variants

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36576064 Hey @ScrapCodes I had some more feedback on documentation and the build stuff. Instead of putting a bunch of comments I just made this pull request into your branch. Do yo

[GitHub] spark pull request: [java8API] SPARK-964 Investigate the potential...

2014-03-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/17#issuecomment-36576111 I think once those are in this is basically ready to go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-03 Thread DB Tsai
Hi Deb, > a. OWL-QN for solving L1 natively in BFGS Based on what I saw from https://github.com/tjhunter/scalanlp-core/blob/master/learn/src/main/scala/breeze/optimize/OWLQN.scala , it seems that it's not difficult to implement OWL-QN once LBFGS is done. > > b. Bound constraints in BFGS : I saw

[GitHub] spark pull request: Remove broken/unused Connection.getChunkFIFO m...

2014-03-03 Thread kayousterhout
GitHub user kayousterhout opened a pull request: https://github.com/apache/spark/pull/69 Remove broken/unused Connection.getChunkFIFO method. This method appears to be broken -- since it never removes anything from messages, and it adds new messages to it, the while loop is a

[GitHub] spark pull request: Remove the remoteFetchTime metric.

2014-03-03 Thread kayousterhout
Github user kayousterhout commented on the pull request: https://github.com/apache/spark/pull/62#issuecomment-36577709 I've merged this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Remove the remoteFetchTime metric.

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/62 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36578345 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36578346 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12973/ --- If your project i

[GitHub] spark pull request: Remove broken/unused Connection.getChunkFIFO m...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/69#issuecomment-36578376 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Remove broken/unused Connection.getChunkFIFO m...

2014-03-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/69#issuecomment-36578374 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10237674 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -847,6 +847,8 @@ class SparkContext( partitions: Seq[Int], allo

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-03-03 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36579672 I've merged this, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10237907 --- Diff: core/src/main/scala/org/apache/spark/util/FileLogger.scala --- @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) unde

[GitHub] spark pull request: Remove broken/unused Connection.getChunkFIFO m...

2014-03-03 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/69#issuecomment-36580142 @tdas should take a look at this actually, I think it was his code. But yes there's no reason to keep FIFO. --- If your project is set up for it, you can reply to this emai

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/50#issuecomment-36580259 Alright, sounds good. Looking forward to it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-972] Added detailed callsite info for V...

2014-03-03 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/34#issuecomment-36580331 Sure, named tuple sounds good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10238309 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io.{FileIn

[GitHub] spark pull request: [WIP] [SPARK-1132] Persisting Web UI through r...

2014-03-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/42#discussion_r10238385 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -17,38 +17,80 @@ package org.apache.spark.ui +import java.io.{FileIn

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-03-03 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/36 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238815 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -71,10 +71,21 @@ private[spark] class CacheManager(blockManager: BlockManager) exten

[GitHub] spark pull request: Patch for SPARK-942

2014-03-03 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/50#discussion_r10238870 --- Diff: core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala --- @@ -23,9 +23,27 @@ import java.nio.ByteBuffer import org.apache.spark

  1   2   3   >