Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-02 Thread DB Tsai
Hi Deb, The PR is here https://github.com/apache/spark/pull/53 Hi Evan, I think we need to refactor the optimization methods and also the way we write algorithms. For example, if I want to use the new optimization method in LogisticRegression.scala, I need to implement LogisticRegressionWithLBF

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36448947 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36448975 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36448976 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36448973 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36448974 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36448998 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36448999 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12947/ --- If your pr

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36449005 Hi guys, I think it is better to make sure Spark doesn't hang when an incorrect partition index is given, because there will be other code paths to run a job. Given

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449019 Thanks. I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36449037 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36449036 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449064 Actually I'm having trouble merging this. I think it's because your git commit doesn't actually have any author information. Do you mind fixing that? cc @pwendell here

[GitHub] spark pull request: Updated more links in documentation

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/23#issuecomment-36449072 #2 was actually merged. @jyotiska do you mind closing this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449083 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449095 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Add Shortest-path computations to graphx.lib w...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/10#issuecomment-36449099 @ankurdave --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449096 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36449165 Thanks. I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-03-02 Thread dbtsai
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36449182 Do I need to close it? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread ngbinh
Github user ngbinh commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449199 just add Author to both PR and the commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project do

[GitHub] spark pull request: Merge the old sbt-launch-lib.bash with the new...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/14#issuecomment-36449214 I've merged this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449234 Sorry @ngbinh you misunderstood me. I think the problem is the git commit metadata doesn't actually contain the author information. It could be that the email or the author in

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/40#issuecomment-36449243 I think the asf git bot will close this once the change is sync-ed on github. If it doesn't get closed tomorrow morning, please close this manually. Thanks! --- If your pr

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36449283 Yes, add too much require statements is unwise.We guarantee throw an error when appropriate, and the rest to the developer to resolve. --- If your project is set up for it,

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread ngbinh
Github user ngbinh commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449390 Yep, found the problem now. Should be fixed! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/41#issuecomment-36449397 Thanks. I've merged this! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this f

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449554 Thanks. I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fe

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36449552 Thanks @pwendell I merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled

[GitHub] spark pull request: Initialized the regVal for first iteration in ...

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/40 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Merge the old sbt-launch-lib.bash with the new...

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/51 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Update io.netty from 4.0.13 Final to 4.0.17.Fi...

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/41 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36449818 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449817 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Remove remaining references to incubation

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/51#issuecomment-36449820 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12948/ --- If your project i

[GitHub] spark pull request: SPARK-1137: Make ZK PersistenceEngine not cras...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4#issuecomment-36449819 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12950/ --- If your project is

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36449823 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12949/ --- If your project i

[GitHub] spark pull request: SPARK-1157 L-BFGS Optimizer based on L-BFGS Ja...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/53#issuecomment-36449822 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Updated more links in documentation

2014-03-02 Thread jyotiska
Github user jyotiska closed the pull request at: https://github.com/apache/spark/pull/23 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36450712 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36450713 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36451660 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12951/ --- If your project i

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36451659 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36453937 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36453936 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36453964 Done, rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36455220 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12952/ --- If your project i

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36455219 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Add Shortest-path computations to graphx.lib w...

2014-03-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/10#issuecomment-36460773 @andy327 would you mind making a JIRA for this? thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If y

Re: MLLib - Thoughts about refactoring Updater for LBFGS?

2014-03-02 Thread Debasish Das
Hi DB, 1. Could you point to the BFGS repositories used to publish artifacts to maven central ? What's the best way to add changes to it ? I fork the repo at my github ? Basically as I mentioned before I need to add lbfgs-b, orthant wise for L1 handling and few variants of line search to lbfgs...

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/54 Ignore RateLimitedOutputStreamSuite for now. This test has been flaky. We can re-enable it after @tdas has a chance to look at it. You can merge this pull request into a Git repository by running:

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/54#issuecomment-36467696 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/54#issuecomment-36467697 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/54#issuecomment-36469429 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12953/ --- If your project i

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/54#issuecomment-36469428 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/54#issuecomment-36469694 Sounds good, merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this fea

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/32#issuecomment-36469726 Thanks, merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

Re: New blog post on Spark + Parquet + Scrooge

2014-03-02 Thread Matei Zaharia
Thanks for posting; I’ve added it to http://spark.apache.org/documentation.html. Matei On Feb 28, 2014, at 1:00 PM, Evan Chan wrote: > http://engineering.ooyala.com/blog/using-parquet-and-scrooge-spark > > Enjoy! > > By the way, I was not able to subscribe to the user-digest list for > some r

[GitHub] spark pull request: SPARK-1145: Memory mapping with many small blo...

2014-03-02 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/43#discussion_r10198184 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskStore.scala --- @@ -84,12 +84,27 @@ private class DiskStore(blockManager: BlockManager, diskManager: D

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10198205 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -950,6 +952,8 @@ class SparkContext( resultHandler: (Int, U) => Unit,

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471158 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: Ignore RateLimitedOutputStreamSuite for now.

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/54 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: SPARK-1084.2 (resubmitted)

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/32 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: SPARK-1121: Include avro for yarn-alpha builds

2014-03-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/49#issuecomment-36471394 Hey guys, I tried building for YARN with 2.2.0 and 0.23.9 and seems like it's working. I'd like to get this in as a build fix, but please re-open SPARK-1121 if you have an

[GitHub] spark pull request: Added a unit test for PairRDDFunctions.lookup

2014-03-02 Thread xoltar
Github user xoltar commented on the pull request: https://github.com/apache/spark/pull/36#issuecomment-36471447 Can someone please merge this? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does n

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471486 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471485 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471529 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12954/ --- If your pr

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471528 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: SPARK-1102: Create a saveAsNewAPIHadoopDataset...

2014-03-02 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/12#issuecomment-36471736 exceed with 5 charssorry.fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-1121: Include avro for yarn-alpha builds

2014-03-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/49 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enable

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/55 SPARK-1158: Fix flaky RateLimitedOutputStreamSuite. There was actually a problem with the RateLimitedOutputStream implementation where the first second doesn't write anything because of integer rounding

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36473419 @tdas @pwendell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36473443 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36473445 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Add Shortest-path computations to graphx.lib w...

2014-03-02 Thread andy327
Github user andy327 commented on the pull request: https://github.com/apache/spark/pull/10#issuecomment-36473467 See JIRA [SPARK-1159]: https://spark-project.atlassian.net/browse/SPARK-1159 --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36473499 Also I think we should reduce the time to run this 4 sec test, but that's for another PR ... --- If your project is set up for it, you can reply to this email and have your r

Re: Development methodology

2014-03-02 Thread Debasish Das
Hi Reynold, I checked and atlassian stash also has pull request feature... In the spark README it says: "Contributions via GitHub pull requests are gladly accepted from their original author." What happens if a spark pull requests comes from stash ? will you guys accept it or all pull requests h

Re: Development methodology

2014-03-02 Thread Reynold Xin
Hi Deb, I am not sure how you can create a pull request coming from stash. Maybe I am not understanding this correctly, but there are only two official Spark repositories: 1. Apache git 2. Github On Sun, Mar 2, 2014 at 4:41 PM, Debasish Das wrote: > Hi Reynold, > > I checked and atlassian sta

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36474899 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12955/ --- If your project i

[GitHub] spark pull request: SPARK-1158: Fix flaky RateLimitedOutputStreamS...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/55#issuecomment-36474898 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/56 Add Jekyll tag to isolate "production-only" doc components. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pwendell/spark jekyll-prod Alternati

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/57 Add Jekyll tag to isolate "production-only" doc components. (0.9 version) You can merge this pull request into a Git repository by running: $ git pull https://github.com/pwendell/spark jekyll-p

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread witgo
Github user witgo commented on a diff in the pull request: https://github.com/apache/spark/pull/44#discussion_r10199454 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -950,6 +952,8 @@ class SparkContext( resultHandler: (Int, U) => Unit,

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/56#issuecomment-36476860 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/57#issuecomment-36476859 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/57#issuecomment-36476857 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/56#issuecomment-36476861 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/57#issuecomment-36476906 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12956/ --- If your pr

[GitHub] spark pull request: Add Jekyll tag to isolate "production-only" do...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/57#issuecomment-36476905 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36477087 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: fix #SPARK-1149 Bad partitioners can cause Spa...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/44#issuecomment-36477086 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-1156: allow user to login into a cluster...

2014-03-02 Thread CodingCat
GitHub user CodingCat opened a pull request: https://github.com/apache/spark/pull/58 SPARK-1156: allow user to login into a cluster without slaves Reported in https://spark-project.atlassian.net/browse/SPARK-1159 The current spark-ec2 script doesn't allow user to login to a

[GitHub] spark pull request: SPARK-1156: allow user to login into a cluster...

2014-03-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/58#issuecomment-36477187 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your proje

[GitHub] spark pull request: SPARK-1156: allow user to login into a cluster...

2014-03-02 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/58#issuecomment-36477477 Reported in [SPARK-1156](https://spark-project.atlassian.net/browse/SPARK-1156), not SPARK-1159. :) --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-1156: allow user to login into a cluster...

2014-03-02 Thread CodingCat
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/58#issuecomment-36477517 oh, fixed, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark pull request: SPARK-1166: clean vpc_id if the group was just...

2014-03-02 Thread CodingCat
GitHub user CodingCat opened a pull request: https://github.com/apache/spark/pull/59 SPARK-1166: clean vpc_id if the group was just now created Reported in https://spark-project.atlassian.net/browse/SPARK-1166 In some very weird situation (when new created group master_group

[GitHub] spark pull request: SPARK-1156: allow user to login into a cluster...

2014-03-02 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/58#issuecomment-36478558 Just curious -- Does this start any of the services correctly ? I think HDFS, Spark won't have any worker nodes in this mode, so the only thing that will work is Spark wit

  1   2   >