+1 support siddhi as a flink operator on DataStream, siddhi support rich CEP features other than pattern match, also support extensible snapshot/restore interface for fault-tolerance which should be easy to integrate with flink's state management (tested with some draft code), we (apache eagle community) use siddhi a lot to integrate with Storm and Spark for real-time monitoring cases. And siddhi's community is active as well in releasing, bug fixing, and accepting external contribution etc.
- Hao On Mon, Aug 29, 2016 at 5:21 PM, Till Rohrmann <trohrm...@apache.org> wrote: > Hi Aparup, > > I haven't looked in detail at Siddhi's internals especially the way it > handles distributed execution. I've only seen that it uses Hazelcast for a > distributed in-memory cache. If a distributed cache for the communication > between instances of the CEP operator is needed, then you would have to > integrate that as well. Otherwise, if the state stored in the operators is > disjunct, then you should be good to go with Flink's state abstraction. > > If you want to work on the integration, then it's best to open a JIRA > issue and describe a little bit the way the integration could look like. > > Cheers, > Till > > On Mon, Aug 29, 2016 at 10:59 AM, Stephan Ewen <se...@apache.org> wrote: > >> Nice idea! >> >> If you look at the current CEP library, it is simply a custom operator. >> Often, you can even get away with a custom FlatMapFunction that uses >> state: https://ci.apache.org/projects/flink/flink-docs-master/dev/ >> state.html#using-the-keyvalue-state-interface >> >> Stephan >> >> >> On Mon, Aug 29, 2016 at 10:09 AM, Aparup Banerjee (apbanerj) < >> apban...@cisco.com> wrote: >> >>> I think siddhi is a fairly matured CEP library. I am thinking it should >>> co-exist with existing CEP library. My thinking is we should be able to use >>> Siddhi QL/ Siddhi Patterns on top of flink data streams. This can co-exist >>> naturally with existing Java / Scala based Flink CEP Library. I am still >>> reading up on Flink internals – but at high level I am thinking about a new >>> Flink operator on DataStream for this. Thoughts ? >>> >>> Thanks, >>> Aparup >>> >>> From: Chesnay Schepler <ches...@apache.org> >>> Reply-To: "user@flink.apache.org" <user@flink.apache.org> >>> Date: Monday, August 29, 2016 at 12:35 AM >>> To: "user@flink.apache.org" <user@flink.apache.org> >>> Subject: Re: Apache siddhi into Flink >>> >>> Hello Aparup, >>> >>> could you provide more information about Siddhi? How mature is it; how >>> is the community? How does it compare to the Flink's CEP library? >>> >>> How should this integration look like? Are you proposing to replace the >>> current CEP library, or will they co-exist with different use-cases for >>> each? >>> >>> If we used Siddhi in Flink, how exactly would Flink's runtime be >>> involved in the processing? >>> >>> Regards, >>> Chesnay >>> >>> On 28.08.2016 23:21, Aparup Banerjee (apbanerj) wrote: >>> >>> Sorry for the semantic difference. >>> >>> >>> >>> On Aug 28, 2016, at 12:05 PM, Trevor Grant < <trevor.d.gr...@gmail.com> >>> trevor.d.gr...@gmail.com> wrote: >>> >>> Thank you for confirming Hao, >>> >>> Aparup, please don't refer to it as "Apache Siddhi", that is >>> misleading. >>> >>> >>> Trevor Grant >>> Data Scientist >>> https://github.com/rawkintrevo >>> http://stackexchange.com/users/3002022/rawkintrevo >>> http://trevorgrant.org >>> >>> *"Fortunate is he, who is able to know the causes of things." -Virgil* >>> >>> >>> On Sun, Aug 28, 2016 at 10:50 AM, Hao Chen <h...@apache.org> wrote: >>> >>>> Siddhi is not apache project, but licensed under apache license v2, >>>> being open sourced and maintained by wso2. >>>> >>>> - Hao >>>> >>>> On Sun, Aug 28, 2016 at 11:11 PM, Trevor Grant < >>>> trevor.d.gr...@gmail.com> wrote: >>>> >>>>> Aparup, >>>>> >>>>> Was Siddhi recently added as an incubator project? I can't find it in >>>>> the project directory or or on github.com/apache. The closest thing >>>>> I can find is this: <https://github.com/wso2/siddhi> >>>>> https://github.com/wso2/siddhi >>>>> >>>>> tg >>>>> >>>>> >>>>> >>>>> >>>>> Trevor Grant >>>>> Data Scientist >>>>> <https://github.com/rawkintrevo>https://github.com/rawkintrevo >>>>> <http://stackexchange.com/users/3002022/rawkintrevo> >>>>> http://stackexchange.com/users/3002022/rawkintrevo >>>>> <http://trevorgrant.org>http://trevorgrant.org >>>>> >>>>> *"Fortunate is he, who is able to know the causes of things." -Virgil* >>>>> >>>>> >>>>> On Sat, Aug 27, 2016 at 5:36 PM, Chen Qin <qinnc...@gmail.com> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> >>>>>> On Aug 26, 2016, at 11:23 PM, Aparup Banerjee (apbanerj) < >>>>>> <apban...@cisco.com>apban...@cisco.com> wrote: >>>>>> >>>>>> Hi- >>>>>> >>>>>> Has anyone looked into embedding apache siddhi into Flink. >>>>>> >>>>>> Thanks, >>>>>> Aparup >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> >
