Hi, We are evaluating Esper <http://www.espertech.com/> to use as a CEP plugged into Flink.
We would want to use Flink's connected streams to connect our rules and events streams and then invoke Esper CEP in the co-process function to evaluate the rules against the events. Would there be any gotchas if we did this ? --Aarti On Thu, Jul 5, 2018 at 6:37 PM, Fabian Hueske <fhue...@gmail.com> wrote: > Hi, > > > Flink doesn't support connecting multiple streams with heterogeneous > schema > > This is not correct. > Flink is very well able to connect streams with different schema. However, > you cannot union two streams with different schema. > In order to reconfigure an operator with changing rules, you can use > BroadcastProcessFunction or KeyedBroadcastProcessFunction [1]. > > In order to dynamically reconfigure aggregations and windowing, you would > need to implement the processing logic yourself in the process function > using state and timers. > There is no built-in support to reconfigure such operators. > > Best, > Fabian > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.5/dev/stream/state/broadcast_state.html > > > 2018-07-05 14:41 GMT+02:00 Puneet Kinra <puneet.ki...@customercentria.com> > : > >> Hi Aarti >> >> Flink doesn't support connecting multiple streams with heterogeneous >> schema ,you can try the below solution >> >> a) If stream A is sending some events make the output of that as >> String/JsonString. >> >> b) If stream B is sending some events make the output of that as >> String/JsonString. >> >> c) Now Using union function you can connect all the streams & use >> FlatMap or process function to >> evaluate all these streams against your defined rules. >> >> d) For Storing your aggregations and rules you can build your cache layer >> and pass as a argument >> to the constructor of that flatmap. >> >> >> >> >> >> >> >> >> On Mon, Jul 2, 2018 at 2:38 PM, Aarti Gupta <aagu...@qualys.com> wrote: >> >>> Hi, >>> >>> We are currently evaluating Flink to build a real time rule engine that >>> looks at events in a stream and evaluates them against a set of rules. >>> >>> The rules are dynamically configured and can be of three types - >>> 1. Simple Conditions - these require you to look inside a single event. >>> Example, match rule if A happens. >>> 2. Aggregations - these require you to aggregate multiple events. >>> Example, match rule if more than five A's happen. >>> 3. Complex patterns - these require you to look at multiple events and >>> detect patterns. Example, match rule if A happens and then B happens. >>> >>> Since the rules are dynamically configured, we cannot use CEP. >>> >>> As an alternative, we are using connected streams and the CoFlatMap >>> function to store the rules in shared state, and evaluate each incoming >>> event against the stored rules. Implementation is similar to what's >>> outlined here >>> <https://data-artisans.com/blog/bettercloud-dynamic-alerting-apache-flink> >>> . >>> >>> My questions - >>> >>> 1. Since the CoFlatMap function works on a single event, how do we >>> evaluate rules that require aggregations across events. (Match rule if >>> more >>> than 5 A events happen) >>> 2. Since the CoFlatMap function works on a single event, how do we >>> evaluate rules that require pattern detection across events (Match rule >>> if >>> A happens, followed by B). >>> 3. How do you dynamically define a window function. >>> >>> >>> --Aarti >>> >>> >>> -- >>> Aarti Gupta <https://www.linkedin.com/company/qualys> >>> Director, Engineering, Correlation >>> >>> >>> aagu...@qualys.com >>> T >>> >>> >>> Qualys, Inc. – Blog <https://qualys.com/blog> | Community >>> <https://community.qualys.com> | Twitter <https://twitter.com/qualys> >>> >>> >>> <https://www.qualys.com/email-banner> >>> >> >> >> >> -- >> *Cheers * >> >> *Puneet Kinra* >> >> *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com >> <puneet.ki...@customercentria.com>* >> >> *e-mail :puneet.ki...@customercentria.com >> <puneet.ki...@customercentria.com>* >> >> >> > -- Aarti Gupta <https://www.linkedin.com/company/qualys> Director, Engineering, Correlation aagu...@qualys.com T Qualys, Inc. – Blog <https://qualys.com/blog> | Community <https://community.qualys.com> | Twitter <https://twitter.com/qualys> <https://www.qualys.com/email-banner>
