BlockManager WARNINGS and ERRORS

HI all, I started testing my code (https://github.com/salexln/FinalProject_FCM) with the latest Spark available in GitHub, and when I run it I get the following errors: *scala> val clusters = FuzzyCMeans.train(parsedData, 2, 20, 2.0)* 16/03/27 22:24:10 WARN BlockManager: Block rdd_8_0 alre

Re: RDD[Vector] Immutability issue

I see, so in order the RDD to be completely immutable, its content should be immutable as well. And if the content is not immutable, we can change its content, but cannot add / remove data? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/RDD-Vector-I

Re: RDD[Vector] Immutability issue

Hi Yu, thanks for the reply. I only run it in my machine (in order to debug the algorithm) . What other conditions did you have it mind? My whole code is here: https://github.com/salexln/FinalProject_FCM Best, Alex -- View this message in context: http://apache-spark-developers-list.1001551

RDD[Vector] Immutability issue

Hi guys, I know the RDDs are immutable and therefore their value cannot be changed but I see the following behaviour: I wrote an implementation for FuzzyCMeans algorithm and now I'm testing it, so i run the following example: import org.apache.spark.mllib.clustering.FuzzyCMeans import org.apache.s

Re: what is the best way to debug spark / mllib?

Thanks for the response, I have several more questions: *1) you should run zinc incremental compiler* I run "./build/zinc-0.3.9/bin/zinc -scala-home $SCALA_HOME -nailed -start" but the compilation time of "build/mvn -DskipTests package' is still about 9 mins. Is this normal? *2) if you want brea

what is the best way to debug spark / mllib?

Hi guys, I'm debugging my code in mllib/clustering but i'm not sure i'm doing it the best way: I build my changes in mllib using "build/mvn -DskipTests package" and then running invoking my code using "./bin/spark-shell" My two main issues: 1) After each change the build (build/mvn -DskipTests p

Re: latest Spark build error

One more question: Is there a way only to build the MLlib using command line? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/latest-Spark-build-error-tp15782p15794.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. -

Re: latest Spark build error

Updating Maven version to 3.3.9 solved the issue Thanks everyone! -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/latest-Spark-build-error-tp15782p15787.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

latest Spark build error

Hi all, I'm getting build error when trying to build a clean version of latest Spark. I did the following 1) git clone https://github.com/apache/spark.git 2) build/mvn -DskipTests clean package But I get the following error: Spark Project Parent POM .. FAILURE [2.338s]

FuzzyCMeans Implementation

Hi guys, I'm working on implementation of FuzzyCMeans (https://issues.apache.org/jira/browse/SPARK-2344), and wanted your thought on whether should FuzzyCMeans class inherit from KMeans? On the one hand, they have a lot of in common, but on the other hand, other algorithms based on KMeans (Bisecti

JIRA 2344 status (Fuzzy C-Means)

Hi guys, Does anyone know what is the status of this issue? (https://issues.apache.org/jira/browse/SPARK-2344) Beniamino was supposed to add an implementation but it seems that his code was removed from Github (https://github.com/bdelpizzo/mllib-extension/blob/master/clustering/FCM.scala) and h

Matix operations in Scala \ Spark

Hi guys, I'm working on the implementation of the FuzzyCMeans algorithm (Jira https://issues.apache.org/jira/browse/SPARK-2344) and I need to use some operations on Matrices (norm & subtraction) I could not find any Scala\ Spark Matrix class that will support these actions. Should I implement th

Re: Contributing to MLlib

thanks for the input. at the moment , I don't have any code commits yet. I wanted to discuss the algorithm implementation prior to the code submission. (never work with Git\ GutHub - so I hope this isn't very basic stuff) -- View this message in context: http://apache-spark-developer

Re: Contributing to MLlib

I opened a JIRA (https://issues.apache.org/jira/browse/SPARK-2344) and a pull request for this (https://github.com/salexln/spark/pull/1) -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7158.html Sent from the Apache Spark

Re: Contributing to MLlib

thanks for the response ! that's is exactly the way i wanted to implement it :) I will create JIRA ticket and a request. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7157.html Sent from the Apache Spark Developers List

Re: Contributing to MLlib

guys??? anyone??? -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Contributing-to-MLlib-tp7125p7155.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Contributing to MLlib

Hi guys, I'm new to Spark & MLlib and this may be a dumb question, but still As part of my M.Sc project, i'm working on implementation of Fuzzy C-means (FCM) algorithm in MLlib. FCM has many things in common with K - Means algorithm, which is already implemented, and I wanted to know whether