I'm afraid that Flink CEP is built rather for lots of data on a couple of rules.
Using any kind of other library would push down the state management and data distribution onto you and negate the main idea of using CEP in the first place. The question is if some of the patterns could be aggregated in a way that fewer operators overall are needed. I was assuming that this kind of optimization already happens but maybe it's insufficient. Could you share your patterns (in an anonymous way)? On Fri, Aug 20, 2021 at 1:46 AM Tejas B <tejasub1...@gmail.com> wrote: > Hi, > Here's our use case : > We are planning to build a rule based engine on top of flink with huge > number of rules(1000s). the rules could be stateless or stateful. > Example stateless rule is : A.id = 3 && A.name = 'abc' || A.color = red. > Example stateful rule is : A is event.id =3, B is event.name = 'abc', C > is event.color = red and we are looking for pattern AB*C over time window > of 1 hour. > > Now we have tried to use flink CEP for this purpose and program crashed > because of lot of threads. The explanation is : every application of > CEP.pattern creates a new operator in the graph and flink can't support > that many vertices in a graph. > > Other approach could be to use processFunction in flink, but still to run > the rules on events stream you'd have to use some sort of CEP or write your > own. > > My question is, does anybody have any other suggestions on how to achieve > this ? Any other CEPs that integrate and work better with flink (siddhi, > jasper, drools) ? Any experience would be helpful. >
