Re: Few basic spark questions

You could implement the receiver as a Spark Streaming Receiver ; the data received would be available for any streaming applications which operate on DStreams (e.g. Streaming KMeans

Re: Few basic spark questions

Hi, Thanks for all the help. I'm still missing something very basic. If I wont use sparkR, which doesn't support streaming (will use mlib instead as Debasish suggested), and I have my scala receiver working, how the receiver should save the data in memory? I do see the store method, so if i use it

Re: Few basic spark questions

What do you need in sparkR that mllib / ml don't havemost of the basic analysis that you need on stream can be done through mllib components... On Jul 13, 2015 2:35 PM, "Feynman Liang" wrote: > Sorry; I think I may have used poor wording. SparkR will let you use R to > analyze the data, but

Re: Few basic spark questions

Sorry; I think I may have used poor wording. SparkR will let you use R to analyze the data, but it has to be loaded into memory using SparkR (see SparkR DataSources ). You will still have to write a Java receiver to store the dat

Re: Few basic spark questions

Hi Oded, I'm not sure I completely understand your question, but it sounds like you could have the READER receiver produce a DStream which is windowed/processed in Spark Streaming and forEachRDD to do the OUTPUT. However, streaming in SparkR is not currently supported (SPARK-6803

Re: Few basic spark questions

any help / idea will be appreciated :) thanks Regards, Oded Maimon Scene53. On Sun, Jul 12, 2015 at 4:49 PM, Oded Maimon wrote: > Hi All, > we are evaluating spark for real-time analytic. what we are trying to do > is the following: > >- READER APP- use custom receiver to get data from rab