Re: Spark support for Complex Event Processing (CEP)

Sorry for answering delay.. Yes, this is not pure "CEP", but quite close for it or many similar "functionalities". My case is not so easy, because i dont' want to compare against original time schedule of route. I want to compare how close (ITS) system has estimated arrival time to bus sto

Re: Spark support for Complex Event Processing (CEP)

If you’re getting the logs, then it really isn’t CEP unless you consider the event to be the log from the bus. This doesn’t sound like there is a time constraint. Your bus schedule is fairly fixed and changes infrequently. Your bus stops are relatively fixed points. (Within a couple of meters)

Re: Spark support for Complex Event Processing (CEP)

ok like any work you need to start this from a simple model. take one bus only (identified by bus number which is unique). for any bus no N you have two logs LOG A and LOG B and LOG C the coordinator from Central computer that sends estimated time of arrival (ETA) to the bus stops. Pretty simple.

Re: Spark support for Complex Event Processing (CEP)

Hi I try to explain my case .. Situation is not so simple in my logs and solution. There also many types of logs and there are from many sources. They are as csv-format and header line includes names of the columns. This is simplified description of input logs. LOG A's: bus coordinate logs

Re: Spark support for Complex Event Processing (CEP)

Look, you said that you didn’t have continuous data, and you do have continuous data. I just used an analog signal which can be converted. So that you end up with contiguous digital sampling. The point is that you have to consider that micro batches are still batched and you’re adding latency

Re: Spark support for Complex Event Processing (CEP)

Also the point about "First there is this thing called analog signal processing…. Is that continuous enough for you? " I agree that analog signal processing like a sine wave, an AM radio signal – is truly continuous. However, here we are talking about digital data which will always be sent as

Re: Spark support for Complex Event Processing (CEP)

In a commerical (C)EP like say StreamBase, or for example its competitor Apama, the arrival of an input event **immediately** triggers further downstream processing. This is admitadly an asynchronous approach, not a synchronous clock-driven micro-batch approach like Spark's. I suppose if one want

Re: Spark support for Complex Event Processing (CEP)

I don’t. I believe that there have been a couple of hack-a-thons like one done in Chicago a few years back using public transportation data. The first question is what sort of data do you get from the city? I mean it could be as simple as time_stamp, bus_id, route and GPS (x,y). Or they co

Re: Spark support for Complex Event Processing (CEP)

Do you know any good examples how to use Spark streaming in tracking public transportation systems ? Or Storm or some other tool example ? Regards Esa Heikkinen 28.4.2016, 3:16, Michael Segel kirjoitti: Uhm… I think you need to clarify a couple of things… First there is this thing called a

Re: Spark support for Complex Event Processing (CEP)

couple of things. There is no such thing as Continuous Data Streaming as there is no such thing as Continuous Availability. There is such thing as Discrete Data Streaming and High Availability but they reduce the finite unavailability to minimum. In terms of business needs a 5 SIGMA is good eno

Re: Spark support for Complex Event Processing (CEP)

Hi Thanks for the answer. I have developed a log file analyzer for RTPIS (Real Time Passenger Information System) system, where buses drive lines and the system try to estimate the arrival times to the bus stops. There are many different log files (and events) and analyzing situation can be

Re: Spark support for Complex Event Processing (CEP)

please see my other reply Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 27 April 2016 at 10:40, Esa

Re: Spark support for Complex Event Processing (CEP)

Hi I have followed with interest the discussion about CEP and Spark. It is quite close to my research, which is a complex analyzing for log files and "history" data (not actually for real time streams). I have few questions: 1) Is CEP only for (real time) stream data and not for "history" d

Re: Spark support for Complex Event Processing (CEP)

h ---21/04/2016 08:34:08 > pm---Hi, Following example in > > From: Mich Talebzadeh > To: Mario Ds Briggs/India/IBM@IBMIN > Cc: Alonso Isidoro Roman , Luciano Resende < > luckbr1...@gmail.com>, "user @spark" > Date: 21/04/2016 08:34 pm > > Subject: Re:

Re: Spark support for Complex Event Processing (CEP)

atabricks/spark-csv thanks Mario From: Mich Talebzadeh To: Mario Ds Briggs/India/IBM@IBMIN Cc: Alonso Isidoro Roman , Luciano Resende , "user @spark" Date: 21/04/2016 08:34 pm Subject: Re: Spark support for Complex Event Processing (CEP) Hi, Fo

Re: Spark support for Complex Event Processing (CEP)

ch Talebzadeh ---19/04/2016 >12:45:52 am---great stuff Mario. Much appreciated. Mich]Mich >Talebzadeh ---19/04/2016 12:45:52 am---great stuff Mario. Much appreciated. >Mich > >From: Mich Talebzadeh <*mich.talebza...@gmail.com* >> >To: Mario Ds Briggs/In

Re: Spark support for Complex Event Processing (CEP)

f Mario. Much appreciated. Mich From: Mich Talebzadeh To: Mario Ds Briggs/India/IBM@IBMIN Cc: "user @spark" , Luciano Resende < luckbr1...@gmail.com> Date: 19/04/2016 12:45 am Subject: Re: Spark support for Complex Event Processing (CEP) great stuff

Re: Spark support for Complex Event Processing (CEP)

reciated. >> >> >> thanks >> Mario >> >> [image: Inactive hide details for Mich Talebzadeh ---19/04/2016 12:45:52 >> am---great stuff Mario. Much appreciated. Mich]Mich Talebzadeh >> ---19/04/2016 12:45:52 am---great stuff Mario. Much appreciated.

Re: Spark support for Complex Event Processing (CEP)

no Resende < > luckbr1...@gmail.com> > Date: 19/04/2016 12:45 am > Subject: Re: Spark support for Complex Event Processing (CEP) > -- > > > > great stuff Mario. Much appreciated. > > Mich > > Dr Mich Talebzadeh > > Link

Re: Spark support for Complex Event Processing (CEP)

am Subject: Re: Spark support for Complex Event Processing (CEP) great stuff Mario. Much appreciated. Mich Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com On 18 April 2016 at 20:08, Mario

Re: Spark support for Complex Event Processing (CEP)

April 2016 at 20:08, Mario Ds Briggs wrote: > Hey Mich, Luciano > > Will provide links with docs by tomorrow > > thanks > Mario > > - Message from Mich Talebzadeh on Sun, 17 > Apr 2016 19:17:38 +0100 - > > *To:* > Luciano Resende > > *cc:* > &q

Re: Spark support for Complex Event Processing (CEP)

cc: "user @spark" Subject: Re: Spark support for Complex Event Processing (CEP) Thanks Luciano. Appreciated. Regard

Re: Spark support for Complex Event Processing (CEP)

Thanks Luciano. Appreciated. Regards Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com On 17 April 2016 at

Re: Spark support for Complex Event Processing (CEP)

Hi Mitch, I know some folks that were investigating/prototyping on this area, let me see if I can get them to reply here with more details. On Sun, Apr 17, 2016 at 1:54 AM, Mich Talebzadeh wrote: > Hi, > > Has Spark got libraries for CEP using Spark Streaming with Kafka by any > chance? > > I a