Re: SparkR and RDDs

2015-05-27 Thread Andrew Psaltis
Hi Shivaram, Thanks for the details, it is greatly appreciated. Thanks On Wed, May 27, 2015 at 7:25 PM, Shivaram Venkataraman < shiva...@eecs.berkeley.edu> wrote: > Sorry for the delay in getting back on this. So the RDD interface is > private in the 1.4 release but as Alek mentioned you can sti

Re: SparkR and RDDs

2015-05-27 Thread Shivaram Venkataraman
Sorry for the delay in getting back on this. So the RDD interface is private in the 1.4 release but as Alek mentioned you can still use it by prefixing `SparkR:::`. Regarding design direction -- there are two JIRAs which cover major features we plan to work on for 1.5. SPARK-6805 tracks porting hi

Re: SparkR and RDDs

2015-05-26 Thread Reynold Xin
You definitely don't want to implement kmeans in R, since it would be very slow. Just providing R wrappers for the MLlib implementation is the way to go. I believe one of the major items in SparkR next is the MLlib wrappers. On Tue, May 26, 2015 at 7:46 AM, Andrew Psaltis wrote: > Hi Alek, > T

Re: SparkR and RDDs

2015-05-26 Thread Andrew Psaltis
Hi Alek, Thanks for the info. You are correct ,that using the three colons does work. Admittedly I am a R novice, but since the three colons is used to access hidden methods, it seems pretty dirty. Can someone shed light on the design direction being taken with SparkR? Should I really be accessing

Re: SparkR and RDDs

2015-05-26 Thread Eskilson,Aleksander
>From the changes to the namespace file, that appears to be correct, all >methods of the RDD API have been made private, which in R means that you may >still access them by using the namespace prefix SparkR with three colons, e.g. >SparkR:::func(foo, bar). So a starting place for porting old Sp