Apache Spark and Swift object store

Hello everyone, I would like to initiate discussion about integration Apache Spark and Openstack Swift. (https://issues.apache.org/jira/browse/SPARK-938 was created while ago) I created a patch (https://github.com/apache/spark/pull/1010) that provides initial information how to connect Swift a

MIMA Compatiblity Checks

Hey All, Some people may have noticed PR failures due to binary compatibility checks. We've had these enabled in several of the sub-modules since the 0.9.0 release but we've turned them on in Spark core post 1.0.0 which has much higher churn. The checks are based on the "migration manager" tool f

Re: Strange problem with saveAsTextFile after upgrade Spark 0.9.0->1.0.0

Moving over to the dev list, as this isn't a user-scope issue. I just ran into this issue with the missing saveAsTestFile, and here's a little additional information: - Code ported from 0.9.1 up to 1.0.0; works with local[n] in both cases. - Driver built as an uberjar via Maven. - Deployed to sma

Re: Strange problem with saveAsTextFile after upgrade Spark 0.9.0->1.0.0

Paul, Could you give the version of Java that you are building with and the version of Java you are running with? Are they the same? Just off the cuff, I wonder if this is related to: https://issues.apache.org/jira/browse/SPARK-1520 If it is, it could appear that certain functions are not in the

Re: Strange problem with saveAsTextFile after upgrade Spark 0.9.0->1.0.0

Also I should add - thanks for taking time to help narrow this down! On Sun, Jun 8, 2014 at 1:02 PM, Patrick Wendell wrote: > Paul, > > Could you give the version of Java that you are building with and the > version of Java you are running with? Are they the same? > > Just off the cuff, I wonder

Re: Strange problem with saveAsTextFile after upgrade Spark 0.9.0->1.0.0

I suspect Patrick is right about the cause. The Maven artifact that was released does contain this class (phew) http://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-core_2.10%7C1.0.0%7Cjar As to the hadoop1 / hadoop2 artifact question -- agree that is often done. Here the working t

Re: Strange problem with saveAsTextFile after upgrade Spark 0.9.0->1.0.0

Okay I think I've isolated this a bit more. Let's discuss over on the JIRA: https://issues.apache.org/jira/browse/SPARK-2075 On Sun, Jun 8, 2014 at 1:16 PM, Paul Brown wrote: > > Hi, Patrick -- > > Java 7 on the development machines: > > » java -version > 1 ↵ > java version "1.7.0_51" > Java(TM)