Re: Few basic spark questions

You could implement the receiver as a Spark Streaming Receiver ; the data received would be available for any streaming applications which operate on DStreams (e.g. Streaming KMeans

Re: Few basic spark questions

Hi, Thanks for all the help. I'm still missing something very basic. If I wont use sparkR, which doesn't support streaming (will use mlib instead as Debasish suggested), and I have my scala receiver working, how the receiver should save the data in memory? I do see the store method, so if i use it

Re: Few basic spark questions

What do you need in sparkR that mllib / ml don't havemost of the basic analysis that you need on stream can be done through mllib components... On Jul 13, 2015 2:35 PM, "Feynman Liang" wrote: > Sorry; I think I may have used poor wording. SparkR will let you use R to > analyze the data, but

Re: Few basic spark questions

Sorry; I think I may have used poor wording. SparkR will let you use R to analyze the data, but it has to be loaded into memory using SparkR (see SparkR DataSources ). You will still have to write a Java receiver to store the dat

Re: Few basic spark questions

Hi Oded, I'm not sure I completely understand your question, but it sounds like you could have the READER receiver produce a DStream which is windowed/processed in Spark Streaming and forEachRDD to do the OUTPUT. However, streaming in SparkR is not currently supported (SPARK-6803

Re: Few basic spark questions

any help / idea will be appreciated :) thanks Regards, Oded Maimon Scene53. On Sun, Jul 12, 2015 at 4:49 PM, Oded Maimon wrote: > Hi All, > we are evaluating spark for real-time analytic. what we are trying to do > is the following: > >- READER APP- use custom receiver to get data from rab

Few basic spark questions

Hi All, we are evaluating spark for real-time analytic. what we are trying to do is the following: - READER APP- use custom receiver to get data from rabbitmq (written in scala) - ANALYZER APP - use spark R application to read the data (windowed), analyze it every minute and save the r

Re: Spark Questions

Thanks for your answers Shuo Xiang and Aaron Davidson! Regards, -- *Gonzalo Zarza* | PhD in High-Performance Computing | Big-Data Specialist | *GLOBANT* | AR: +54 11 4109 1700 ext. 15494 | US: +1 877 215 5230 ext. 15494 | [image: Facebook] [image: Twitter]

Re: Spark Questions

I am not entirely certain I understand your questions, but let me assume you are mostly interested in SparkSQL and are thinking about your problem in terms of SQL-like tables. 1. Shuo Xiang mentioned Spark partitioning strategies, but in case you are talking about data partitioning or sharding as

Re: Spark Questions

For your first question, the partitioning strategy can be tuned by applying different partitioner. You can use existing ones such as HashPartitioner or write your own.See this link( http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/matei-zaharia-amp-camp-2012-advanced-spark.pdf) for some instr

Spark Questions

Hi all, We've been evaluating Spark for a long-term project. Although we've been reading several topics in forum, any hints on the following topics we'll be extremely welcomed: 1. Which are the data partition strategies available in Spark? How configurable are these strategies? 2. How would be t

RE: Basic Scala and Spark questions

Subject: RE: Basic Scala and Spark questions Date: Tue, 24 Jun 2014 20:06:04 + Hello Tilak, 1. I get error Not found: type RDD error. Can someone please tell me which jars do I need to add as external jars and what dhoulf I add iunder import statements so that this error will go away

RE: Basic Scala and Spark questions

To: u...@spark.incubator.apache.org Subject: Basic Scala and Spark questions Hi All, I am new so Scala and Spark. I have a basic question. I have the following import statements in my Scala program. I want to pass my function (printScore) to Spark. It will compare a string i

RE: Basic Scala and Spark questions

sic Scala and Spark questions Date: Mon, 23 Jun 2014 10:38:04 -0700 Hi All,I am new so Scala and Spark. I have a basic question. I have the following import statements in my Scala program. I want to pass my function (printScore) to Spark. It will compare a string imp

Basic Scala and Spark questions

Hi All,I am new so Scala and Spark. I have a basic question. I have the following import statements in my Scala program. I want to pass my function (printScore) to Spark. It will compare a string import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import o