This is one of my challenges too-
1. The JavaScript rules are only applicable inside one operator (next,
followedBy, notFollowedBy). And the JavaScript rules can apply to only the
event in that operator. I make it a little more dynamic by creating a Rules
HashMap and add rules with the names "first", "next", "followedBy1" (same
as the ones I use for the pattern operator). This way the rules attached to
a particular operator can be easily changed via a connected stream.
I think the feature where other events in the pattern are accessible is
being added. Currently you can only look inside one event in the "where"
clause attached to a CEP pattern operator. For example, if I check two
consecutive credit card events for a user to calculate the straight line
distance between then to divide by the difference in time I cannot do that
unless I fire the pattern for every pair and check this condition in the
PatternStream's select operator where all the events are accessible.
2. The second problem I have is that I cannot change the rules applied to
the pattern stream. For example if I have next.followedBy and I want to add
another followedBy it is a compile time change. The JavaScript engine helps
me with the first issue but this one just needs a recompile unless you have
another Flink Pipeline deployed which can check against that pattern as
well. I am guess at this point you will need to take a SavePoint, stop your
pipeline, redeploy this new pipeline(with a new pattern configuration) and
start again.
I would like to know if there is a cleaner solution but the above is my
fallback.
Sameer
On Tue, Oct 11, 2016 at 5:51 PM, <lg...@yahoo.com> wrote:
> Hi Sameer,
>
> I just replied to the earlier post, but I will copy it here:
>
> We also have the same requirement - we want to allow the user to change
> the matching patterns and have them take effect immediately. I'm wondering
> whether the proposed trigger DSL takes us one step closer:(I don't think it
> solves the problem) or we have to dynamically generate Flink job JAR files
> when the matching rules/patterns are changed and submit them to Flink.
>
> I had thought about using a similar approach, but it is quite restrictive
> because you cannot use the power for Flink CEP engine with this approach.
> For example, I want to be able to use followedBy, next, notFollowedBy (in
> future) operators to detect the patterns and these matching patterns need
> to be user-cofigurable/dynamic/hot deployable. The simple attribute-based
> rules/patterns that you specified can be made dynamic as you mentioned but
> the rules/patterns that use not just the current event attributes, but also
> past events (e.g. followedBy) are much harder to make them dynamic without
> some help from Flink that implements the CEP operators.
>
> - LF
>
>
>
>
> ------------------------------
> *From:* Sameer W <sam...@axiomine.com>
> *To:* "user@flink.apache.org" <user@flink.apache.org>
> *Sent:* Tuesday, October 11, 2016 2:23 PM
> *Subject:* Re: What is the best way to load/add patterns dynamically (at
> runtime) with Flink?
>
> I have used a JavaScript engine in my CEP to evaluate my patterns. Each
> event is a list of named attributes (HashMap like). And event is attached
> to a list of rules expressed as JavaScript code (See example below with one
> rule but I can match as many rules). The rules are distributed over a
> connected stream which allow it to change over time. This is how I do it to
> keep my rules dynamic. If someone has a better way I would love to hear it
> as well.
>
>
> private transient ScriptEngineManager factory = new ScriptEngineManager();
> private transient ScriptEngine engine = factory.getEngineByName("
> JavaScript");
> /*Inside open*/
> factory = new ScriptEngineManager();
>
> /*Close open*/
>
> /*Inside my operator*/
> engine = factory.getEngineByName("JavaScript");
> engine.put("evt", value.f1); //value.f1 contains a JSON version of my
> HashMap of attributes
> engine.eval(value.f2.rule); //f2.last contains the rule which is evaluated
> by the JavaScript Engine
> /*
> Sample JavaScript contained in the call - engine.eval(value.f2.rule); is
> shown below (not the "evt" variable in the JavaScript and the the preceding
> line - engine.put("evt", value.f1);
>
> *var evt=JSON.parse(evt);var result = evt.temperature>50 &&
> evt.pressure<900*
> */
> boolean ret = (boolean)engine.get("result");
>
> if(ret) /*Rule is Matched*/
>
>
>
> > On Oct 11, 2016, at 5:01 PM, PedroMrChaves <pedro.mr.cha...@gmail.com>
> wrote:
> >
> > Hello,
> >
> > I am new to Apache Flink and am trying to build a CEP using Flink's API.
> One
> > of the requirements is the ability to add/change patterns at runtime for
> > anomaly detection (maintaining the systems availability). Any Ideas of
> how
> > could I do that?
> >
> > For instance, If I have a stream of security events (accesses,
> > authentications ,etc.) and a pattern for detecting anomalies I would
> like to
> > be able to change that pattern parameters, for instance instead of
> detecting
> > the occurrence of events A->B->C I would like to change the condition on
> B
> > to B’ in order to have a new rule. Moreover, I would like to be able to
> > create new patterns dynamically as new use cases arise.
> >
> > Best Regards,
> > Pedro Chaves
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/What-is-the-
> best-way-to-load-add-patterns-dynamically-at-runtime-with-
> Flink-tp9461.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>
>
