Windows CI infra

I've seen other projects use Appveyor for CI on Windows. Has anyone used them before? I've seen on more than one occasion something break on Windows without us knowing, so it might be worth looking into using something like this if it's relatively straightforward. Nick

RE: Running LocalClusterSparkContext

Thank you so much! It allows me to proceed and Spark started to execute my code. I use sc.textFile and Spark crashes on it: 15/04/03 15:15:21.572 sparkDriver-akka.actor.default-dispatcher-4 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkexecu...@mynetwork.net:5765

Re: Running LocalClusterSparkContext

SPARK-2356 On Fri, Apr 3, 2015 at 1:01 PM, Ulanov, Alexander wrote: > Thanks! It worked. I've updated the PR and now start the test with working > directory in SPARK_HOME > > Now I get: > ERROR Shell: Failed to locate the winutils binary in the hadoop binary path > java.io.IOException: Could not

RE: Running LocalClusterSparkContext

Thanks! It worked. I've updated the PR and now start the test with working directory in SPARK_HOME Now I get: ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. Any ideas? --

Re: Running LocalClusterSparkContext

That looks like another bug on top of 6673; can you point that out in the PR to make sure it's covered? In the meantime, could you try running the command from $SPARK_HOME instead of from inside the mllib directory? On Fri, Apr 3, 2015 at 12:48 PM, Ulanov, Alexander wrote: > Hi Marcelo, > > Than

RE: Running LocalClusterSparkContext

Hi Marcelo, Thank you for quick response! It seems that I get the issue 6673. If I set set SPARK_SCALA_VERSION=2.10 as suggested in https://issues.apache.org/jira/browse/SPARK-6673, I get instead: 15/04/03 12:46:24.510 ExecutorRunner for app-20150403124624-/0 ERROR ExecutorRunner: Error run

Re: Running LocalClusterSparkContext

When was the last time you pulled? That should have been fixed as part of SPARK-6473. Notice latest master suffers from SPARK-6673 on Windows. On Fri, Apr 3, 2015 at 12:21 PM, Ulanov, Alexander wrote: > Hi, > > I am trying to execute unit tests with LocalClusterSparkContext on Windows 7. > I am

Running LocalClusterSparkContext

Hi, I am trying to execute unit tests with LocalClusterSparkContext on Windows 7. I am getting a bunch of error in the log saying that: "Cannot find any assembly build directories." Below is the part from the log where it brakes. Could you suggest what's happening? In addition the application h

Spark PR builder now with dependency reports!

All, recently Spark has updated the PR builder in an attempt to better help the committers understand what, if anything, has changed wrt dependencies for new PR’s. This comes in the form of a fourth bullet posted on Github that I’m sure you’ve noticed when Jenkins is done building and running th

Re: Haskell language Spark support

The PR for integrate SparkR into Spark may help: https://github.com/apache/spark/pull/5096 -- Davies Liu Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Wednesday, March 25, 2015 at 7:35 PM, danilo2 wrote: > Hi! > I'm a haskell developer and I have created many haskell libraries in

extended jenkins downtime, thursday april 9th 7am-noon PDT (moving to anaconda python & more)

welcome to python2.7+, java 8 and more! :) i'll be doing a major upgrade to our build system next thursday morning. here's a quick list of what's going on: * installation of anaconda python on all worker nodes * installation of pypy 2.5.1 (python 2.7) on all nodes * matching installation of p

RE: IntelliJ Runtime error

Thank you, it works with me when I changed the dependencies from provided to compile. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/IntelliJ-Runtime-error-tp11383p11385.html Sent from the Apache Spark Developers List mailing list archive at Nabble.c

RE: IntelliJ Runtime error

You have to change most of dependences in the spark-example model from "provided" to "compile", so you can run the example in Intellij. Yong > Date: Fri, 3 Apr 2015 09:22:13 -0700 > From: eng.sara.must...@gmail.com > To: dev@spark.apache.org > Subject: IntelliJ Runtime error > > Hi, > > I have

IntelliJ Runtime error

Hi, I have built Spark 1.3.0 successfully on IntelliJ IDEA 14, but when i try to SparkPi example under the examples module i face this error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:27)

Wrong initial bias in GraphX SVDPlusPlus?

I believe that in the initialization portion of GraphX SVDPlusPluS, the initialization of biases is incorrect. Specifically, in line https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/SVDPlusPlus.scala#L96 instead of (vd._1, vd._2, msg.get._2 / msg.ge

Re: Unit test logs in Jenkins?

> On 2 Apr 2015, at 19:44, Marcelo Vanzin wrote: > > On Thu, Apr 2, 2015 at 3:01 AM, Steve Loughran wrote: That would be really helpful to debug build failures. The scalatest output isn't all that helpful. >> >> Potentially an issue with the test runner, rather than the tests t