localDF is a pure R data frame and as.vector will work with no problems, as for calling it in the SparkR objects, try calling collect before you call as.vector (or in your case, the algorithms), that should solve your problem.
On Mon, Sep 21, 2015 at 8:48 AM, Ellen Kraffmiller < ellen.kraffmil...@gmail.com> wrote: > Thank you for the link! I was using > http://apache-spark-user-list.1001560.n3.nabble.com/, and I didn't see > replies there. > > Regarding your code example, I'm doing the same thing and successfully > creating the rdd, but the problem is that when I call a clustering > algorithm like amap::hcluster(), I get an error from as.vector() that the > rdd cannot be coerced into a vector. > > On Fri, Sep 18, 2015 at 12:33 PM, Luciano Resende <luckbr1...@gmail.com> > wrote: > >> I see the thread with all the responses on the bottom at mail-archive : >> >> https://www.mail-archive.com/user%40spark.apache.org/msg36882.html >> >> On Fri, Sep 18, 2015 at 7:58 AM, Ellen Kraffmiller < >> ellen.kraffmil...@gmail.com> wrote: >> >>> Thanks for your response. Is there a reason why this thread isn't >>> appearing on the mailing list? So far, I only see my post, with no >>> answers, although I have received 2 answers via email. It would be nice if >>> other people could see these answers as well. >>> >>> On Thu, Sep 17, 2015 at 2:22 AM, Sun, Rui <rui....@intel.com> wrote: >>> >>>> The existing algorithms operating on R data.frame can't simply operate >>>> on SparkR DataFrame. They have to be re-implemented to be based on SparkR >>>> DataFrame API. >>>> >>>> -----Original Message----- >>>> From: ekraffmiller [mailto:ellen.kraffmil...@gmail.com] >>>> Sent: Thursday, September 17, 2015 3:30 AM >>>> To: user@spark.apache.org >>>> Subject: SparkR - calling as.vector() with rdd dataframe causes error >>>> >>>> Hi, >>>> I have a library of clustering algorithms that I'm trying to run in the >>>> SparkR interactive shell. (I am working on a proof of concept for a >>>> document classification tool.) Each algorithm takes a term document matrix >>>> in the form of a dataframe. When I pass the method a local dataframe, the >>>> clustering algorithm works correctly, but when I pass it a spark rdd, it >>>> gives an error trying to coerce the data into a vector. Here is the code, >>>> that I'm calling within SparkR: >>>> >>>> # get matrix from a file >>>> file <- >>>> >>>> "/Applications/spark-1.5.0-bin-hadoop2.6/examples/src/main/resources/matrix.csv" >>>> >>>> #read it into variable >>>> raw_data <- read.csv(file,sep=',',header=FALSE) >>>> >>>> #convert to a local dataframe >>>> localDF = data.frame(raw_data) >>>> >>>> # create the rdd >>>> rdd <- createDataFrame(sqlContext,localDF) >>>> >>>> #call the algorithm with the localDF - this works result <- >>>> galileo(localDF, model='hclust',dist='euclidean',link='ward',K=5) >>>> >>>> #call with the rdd - this produces error result <- galileo(rdd, >>>> model='hclust',dist='euclidean',link='ward',K=5) >>>> >>>> Error in as.vector(data) : >>>> no method for coercing this S4 class to a vector >>>> >>>> >>>> I get the same error if I try to directly call as.vector(rdd) as well. >>>> >>>> Is there a reason why this works for localDF and not rdd? Should I be >>>> doing something else to coerce the object into a vector? >>>> >>>> Thanks, >>>> Ellen >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-calling-as-vector-with-rdd-dataframe-causes-error-tp24717.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For >>>> additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >> >> -- >> Luciano Resende >> http://people.apache.org/~lresende >> http://twitter.com/lresende1975 >> http://lresende.blogspot.com/ >> > > -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
