Re: Flink CEP Resource Utilisation Optimisation

> *发送时间:* Monday, March 27, 2023 12:38:59 AM >> *收件人:* Geng Biao >> *抄送:* user@flink.apache.org >> *主题:* Re: Flink CEP Resource Utilisation Optimisation >> >> Thanks, Geng for the quick and actionable response. >> >> I will definitely try this with Flink v

Re: Flink CEP Resource Utilisation Optimisation

- > *发件人:* Abhishek Singla > *发送时间:* Monday, March 27, 2023 12:38:59 AM > *收件人:* Geng Biao > *抄送:* user@flink.apache.org > *主题:* Re: Flink CEP Resource Utilisation Optimisation > > Thanks, Geng for the quick and actionable response. > > I wil

Re: Flink CEP Resource Utilisation Optimisation

. Best, Biao 获取 Outlook for iOS<https://aka.ms/o0ukef> 发件人: Abhishek Singla 发送时间: Monday, March 27, 2023 12:38:59 AM 收件人: Geng Biao 抄送: user@flink.apache.org 主题: Re: Flink CEP Resource Utilisation Optimisation Thanks, Geng for the quick and actionable respon

Re: Flink CEP Resource Utilisation Optimisation

Thanks, Geng for the quick and actionable response. I will definitely try this with Flink version >= 1.16.0 and get back with the observations. Regarding the checkpoint size issue, my concern is if there is no more state, shouldn't the checkpoint size be way less than 2 GB? I mean I was expecting

Re: Flink CEP Resource Utilisation Optimisation

Hi Abhishek, Thanks for sharing the experiment! As for the performance question, I believe you could give a try on Flink CEP with version >= 1.16.0, which includes the optimization introduced in FLINK-23890. This optimization will reduce lots

Re: Flink CEP Resource Utilisation Optimisation

退订 发自我的iPhone -- Original -- From: Abhishek Singla

Re: Flink CEP Incremental Checkpoint Issue

Hi Yun Tang, Thank you for the response and yes went through some articles which explained rocksdb incremental checkpointing mechanism and makes sense w.r.t. metrics that i am seeing. Regards, Puneet > On 22-Oct-2022, at 1:26 PM, Yun Tang wrote: > > compaction

Re: Flink CEP Incremental Checkpoint Issue

hould care about the checkpoint size too much. Instead, we should care more about the output results. Best Yun Tang From: Martijn Visser Sent: Wednesday, October 19, 2022 22:03 To: Puneet Duggal Cc: user Subject: Re: Flink CEP Incremental Checkpoint Issue Hi,

Re: Flink CEP Incremental Checkpoint Issue

Hi, Given that Flink 1.12 is no longer supported by the community, can you validate this with the latest Flink version? (Currently 1.15). Next to that, the contents of your checkpoints is not only the results of your CEP, but given that you're using Exactly Once also there's internal information n

Re: Flink CEP Incremental Checkpoint Issue

Apologies for the mistake of calculation 120*6*2KB = 1440KB = 1.4MB > On 18-Oct-2022, at 1:35 AM, Puneet Duggal wrote: > > Hi, > > I am working on a use case which uses Flink CEP for pattern detection. > > Flink Version - 1.12.1 > Deployment Mode - Session Mode (Highly Available) > State Back

Re: Flink CEP in PyFlink

Hi Kamil, It still doesn’t support CEP in PyFlink. However, as it supports SQL in PyFlink and so you could take a look at if CEP on SQL [1] could meet your requirements. If so, you could use CEP on SQL in PyFlink. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/

Re: Flink cep checkpoint size

Hi, Dawid. Thanks for replying, happy to know you are working on this. On 2021/07/08 12:14:21, Dawid Wysakowicz wrote: > Hi, > > Sorry for the late reply. > > Indeed I found a couple of problems with clearing the state for short > lived keys. I created a JIRA[1] issue to track it and opened a

Re: Flink cep checkpoint size

Hi, Sorry for the late reply. Indeed I found a couple of problems with clearing the state for short lived keys. I created a JIRA[1] issue to track it and opened a PR (which needs test coverage before it can be merged) with fixes for those. Best, Dawid [1] https://issues.apache.org/jira/browse/

Re: Flink CEP checkpoint size

We did look into fixing it ourselves, but decided that migrating to the datastream api, not using CEP, was more fruitful overall for us. Unfortunately, I don't have a good answer for you. The bug from a non-contributors stand point appears pretty deep in the codebase, but the authors are best ones

Re: Flink CEP checkpoint size

Hi, Jameson Thanks very much for replying , I am really struggling on this. I am using flowId as my keys, which means they will be matched and never use again. This seems like the scenario 2. I didn't know it is not fixed yet. thank you again and do you have any solutions ？ On 2021/07/07 01:47:00

Re: Flink CEP checkpoint size

Hi Li, How big is your keyspace? Had a similar problem which turns out to be scenario 2 in this issue https://issues.apache.org/jira/browse/FLINK-19970. Looks like the bug in scenario 1 got fixed by scenario 2 did not. There's more detail in this thread, http://deprecated-apache-flink-user-ma

Re: Flink cep checkpoint size

Hi, Mohit, Have you figured out any solusions on this problem ? I am now facing the exactly same problem , I was using Flink of version 1.12.0 and I also upgrated it to 1.13.1 but the checkpoint size is still growing. On 2021/06/02 15:45:59, "Singh, Mohit" wrote: > Hi, > > I am facing an i

Re: Flink CEP: can't process PatternStream

Hello, David. Yes, I’m using 1.12. And my code is now working. Thank you very much for your comment. Yuri L. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink CEP: can't process PatternStream (v 1.12, EventTime mode)

Hi, What is exactly the problem? Is it that no patterns are being generated? Usually the problem is in idle parallel instances[1]. You need to have data flowing in each of the parallel instances for a watermark to progress. You can also read about it in the aspect of Kafka's partitions[2]. Best,

Re: Flink CEP: can't process PatternStream

Hi Yuri, Which Flink version are you using? Is it 1.12? In 1.12 we changed the default TimeCharacteristic to EventTime. Therefore you need watermarks and timestamp[1] for your program to work correctly. If you want to apply your pattern in ProcessingTime you can do: PatternStream patternStream =

Re: Flink CEP pattern design question

I think what you ask for is something like timing out greedy[1] quantifier, which is not supported. As a rather dirty workaround you could try sth like: Pattern .begin[Event]("start") .where(_._.getName == "a") .oneOrMore.opt

Re: [flink-cep] - What is the difference between GroupPattern and Pattern?

Hi Jiayi, GroupPattern is used to support patterns like this: A (B C) {3} D. The sub-group (B C) is considered as a GroupPattern. Without it, it's impossible to support patterns like that. You can refer to the GroupITCase for more information. Regards, Dian > 在 2019年2月15日，下午4:10，bupt_ljy 写道：

Re: Flink CEP : Doesn't generate output

Can you provide us a self-contained reproducing example? (preferably as elementary as possible) On 22.01.2019 18:58, dhanuka ranasinghe wrote: Hi All, I have used Flink CEP to filter some events and generate some alerts based on certain conditions. But unfortunately doesn't print any result.

Re: Flink CEP : Doesn't generate output

Thank you for the clarification. On Thu, 24 Jan 2019, 12:44 Dian Fu Hi Dhanuka, > > From the code you shared, it seems that you're using event time. The > processing of elements is triggered by watermark in event time and so you > should define how to generate the watermark, i.e with > DataStream

Re: Flink CEP : Doesn't generate output

Hi Dhanuka, From the code you shared, it seems that you're using event time. The processing of elements is triggered by watermark in event time and so you should define how to generate the watermark, i.e with DataStream.assignTimestampsAndWatermarks Regards, Dian > 在 2019年1月23日，上午1:58，dhanuka

Re: Flink CEP support pattern match involving fields of previous events

Hi Florin, This feature is supported with IterativeCondition since 1.3.0. For questions about API and what features are supported in general please always have a look into documentation[1] first. Best, Dawid [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/libs/cep.html On

Re: [flink-cep] Flick CEP support for the group By operator

Hey, As a matter of fact, you do not need a Flink's CEP library to run the same query. The same functionality can be achieved by simple tumbling window with a “median” aggregate (“median" you would have to implement by your self). https://ci.apache.org/projects/flink/flink-docs-stable/dev/strea

Re: [flink-cep] Flick CEP support for the group By operator

Hi, Yes, sure. Just use CEP on top of KeyedStream. Take a look at `keyBy`: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/ Piotrek > On 23 Nov 2018, at 16:04, Spico Florin wrote: > > Hello! > > I'm using Flink 1.4.2 and I would like to use a group by operat

Re: Flink CEP Watermark Exception

Hi Dawid, Just back in the office. The platform we run on recently announced Flink 1.6.0 support, so we upgraded and haven't seen this problem arise again yet! We believe it could have been the `equals` method falsely matching different records in rare instances, though the upgrade to Flink 1.6.0

Re: Flink CEP Watermark Exception

Hi Austin, Could you provide jobmanagers and taksmanagers logs for a failed run? The exception you've posted is thrown during processing messages, rather than during restoring, but you said it failed to restore checkpoint, how come it processes messages? Could you also describe exact conditions st

Re: Flink CEP Watermark Exception

Hi Dawid, Thank you for your reply. I'm out for the next few days, so I hope you don't mind me cc'ing my team in here. We all really appreciate you and the rest of the people monitoring the mailing list. We've only seen this SharedBuffer problem in production, after sending around 20 GB of data

Re: Flink CEP Watermark Exception

Hi Austin, Could you elaborate a bit more what do you mean by "after a checkpoint fails", what is the reason why checkpoint fails? Would it be possible for you to prepare some reproducible example for that problem? Finally, I would also recommend trying out Flink 1.6.x, as we reworked the underlyi

Re: Flink CEP Watermark Exception

Following up, we are using Flink 1.5.0 and Flink-CEP 2.11. Thanks, Austin On Tue, Oct 30, 2018 at 3:58 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi there, > > > We have a streaming application that uses CEP processing but are getting this > error fairly frequently after a che

Re: Flink CEP window for 1 working day

Hi, I don't think the CEP library is that flexible, but I loop in Kostas (CC) who knows more about it. I'm not exactly sure what you mean by "manipulate" event-time, but I don't think that's necessary. You can implement rules also with state and timers in the ProcessFunction. The function ingests

Re: Flink CEP window for 1 working day

Thanks Fabian, So by non working day, I mean, I have a list of non working day in a year, which I can use to compare. I am very new to Flink and Flick CEP. Initially I thought there is a way to have within(time) value expression dynamically. So now I guess that's not possible. If I understand c

Re: Flink CEP window for 1 working day

Hi, I'm afraid, Flink CEP does not distinguish work days from non-work days. Of course, you could implement the logic in a DataStream program (probably using ProcessFunction). Best, Fabian 2018-03-20 15:44 GMT+01:00 shishal : > I am using flink CEP , and to match a event pattern in given time w

Re: Flink CEP with files and no streams?

you have > any good Scala code examples about this ? > > > > Esa > > > > *From:* Fabian Hueske [mailto:fhue...@gmail.com] > *Sent:* Wednesday, February 7, 2018 11:21 AM > *To:* Esa Heikkinen > *Cc:* user@flink.apache.org > *Subject:* Re: Flink CEP with files an

RE: Flink CEP with files and no streams?

Hi Thanks for the reply, but because I am a newbie with Flink, do you have any good Scala code examples about this ? Esa From: Fabian Hueske [mailto:fhue...@gmail.com] Sent: Wednesday, February 7, 2018 11:21 AM To: Esa Heikkinen Cc: user@flink.apache.org Subject: Re: Flink CEP with files and

Re: Flink CEP with files and no streams?

Hi Esa, you can also read files as a stream. However, you have to be careful in which order you read the files and how you generate watermarks. The easiest approach is to implement a non-parallel source function that reads the files in the right order and generates watermarks. Things become more t

Re: Flink CEP exception during RocksDB update

Hi Varun, The branch I previously sent you has been now merged to the master. So could you try the master and tell us if you see any change in the behavior? Has the problem been fixed, or has the message of the exception changed? Thanks, Kostas > On Jan 29, 2018, at 10:09 AM, Kostas Kloudas

Re: Flink CEP exception during RocksDB update

Hi again Varun, I am investigating the problem you mentioned and I found a bug in the SharedBuffer, but I am not sure if it is the only bug that affects you. Could you please try this branch https://github.com/kl0u/flink/tree/cep-inv and let me know

Re: Flink CEP exception during RocksDB update

Hi Varun, Thanks for taking time to look into it. Could you give a sample input and your pattern to reproduce the problem? That would help a lot at figuring out the cause of the problem. Thanks, Kostas > On Jan 23, 2018, at 5:40 PM, Varun Dhore wrote: > > Hi Kostas, > > I was able to reprodu

Re: Flink CEP exception during RocksDB update

Hi Kostas, I was able to reproduce the error with 1.4.0. After upgrading the cluster to 1.5 snapshot and running through the same data I am still experiencing the same exception. CEP patterns that I am running are using followed by patterns e.g AfBfC. From my experience I was never able to get

Re: Flink CEP exception during RocksDB update

Thanks a lot Varun! Kostas > On Jan 17, 2018, at 9:59 PM, Varun Dhore wrote: > > Thank you Kostas. Since this error is not easily reproducible on my end I’ll > continue testing this and confirm the resolution once I am able to do so. > > Thanks, > Varun > > Sent from my iPhone > > On Jan 1

Re: Flink CEP exception during RocksDB update

Thank you Kostas. Since this error is not easily reproducible on my end I’ll continue testing this and confirm the resolution once I am able to do so. Thanks, Varun Sent from my iPhone > On Jan 15, 2018, at 10:21 AM, Kostas Kloudas > wrote: > > Hi Varun, > > This can be related to this iss

Re: Flink CEP exception during RocksDB update

Hi Varun, This can be related to this issue: https://issues.apache.org/jira/browse/FLINK-8226 which is currently fixed on the master. Could you please try the current master to see if the error persists? Thanks, Kostas > On Jan 15, 2018, at 4

Re: Flink CEP with event time

Yes, because event-time only advances if something makes it advance. Basically. > On 4. Jan 2018, at 11:34, shashank agarwal wrote: > > But this will be wrong in my case. So I have to wait for the results until I > receive next event. > > > > ‌ > > On Thu, Jan 4, 2018 at 3:53 PM, Aljoscha K

Re: Flink CEP with event time

But this will be wrong in my case. So I have to wait for the results until I receive next event. ‌ On Thu, Jan 4, 2018 at 3:53 PM, Aljoscha Krettek wrote: > Think this is actually working as intended, from your earlier description > of when results are produced: When you see Event 1.B, the wa

Re: Flink CEP with event time

Think this is actually working as intended, from your earlier description of when results are produced: When you see Event 1.B, the watermark is not sufficiently advanced to trigger computation, only when you see Event 2.A does the watermark advance and you get a result. This is what I would exp

Re: Flink CEP with event time

@Dawid, I was using 1.3.2, I have checked on 1.4.0 also still facing the same issue. @Aljoscha, I have to cover the case where B can come after A from Kafka. How I can achieve this as Event Time is not working. How should I implement this? A followedBy B. As I am using kafka source and my even

Re: Flink CEP with event time

Hi shashank, What version of flink are you using? Is it possible that you are hitting this issue: https://issues.apache.org/jira/browse/FLINK-7563 ? Watermark semantics in CEP was buggy and events were processed only if its timestamp was lower than current watermark while it should be lower or

Re: Flink CEP with event time

What are the actual timestamps? If your BoundedOutOfOrderness extractor is lagging by 10 seconds then only seeing Event 1.B would not trigger execution. Only the later Event 2.A is sufficiently far ahead to trigger execution, which you actually get. > On 3. Jan 2018, at 17:05, shashank agarwal

Re: Flink CEP with event time

Low Watermark is showing the same value which I am passing in event "1514994744412" for all the tasks related to that stream, (No watermark) is showing for Kafka source in UI. So the pattern is following for CEP A followedBy B : Event 1 - I passed A with origTimestamp X. (Low watermark updated to

Re: Flink CEP with event time

Can you please check what the input watermark of your operations is? There is a metric called "currentLowWatermark" for this. Best, Aljoscha > On 3. Jan 2018, at 15:54, shashank agarwal wrote: > > Actually, In Kafka there are other topics also (around 5-6 topics) I am > consuming particular t

Re: Flink CEP with event time

Actually, In Kafka there are other topics also (around 5-6 topics) I am consuming particular topic 'x' which only contains events. Other topics have different data. I am using two consumers in my program for 2 different topics. in first topic x i am extracting the timestamp from origintimestamp v

Re: Flink CEP with event time

Ok, but will there be events in all Kafka partitions/topics? > On 3. Jan 2018, at 15:33, shashank agarwal wrote: > > Hi, > > Yes, Events will always carry a variable OriginTimestamp which I am using in > the extractor. I have used fallback also in case of data missing will put > System curren

Re: Flink CEP with event time

Hi, Yes, Events will always carry a variable OriginTimestamp which I am using in the extractor. I have used fallback also in case of data missing will put System current millis. Still, it's not printing results. Best, Shashank ‌ On Wed, Jan 3, 2018 at 7:40 PM, Aljoscha Krettek wrote: > H

Re: Flink CEP with event time

Hi, Are all the partitions always carrying data that has advancing timestamps? When using Event-time the Kafka source (and Flink in general) needs to have steady progress in all partitions, otherwise the watermark does not advance, which in turn means that processing will be stalled downstream.

Re: Flink CEP state store

Hi Mahesh, You are right that it based on the Pattern you write there might be a big amount of intermittent states created. Flink CEP library keeps all its State in State Backend. So if you use e.g. RocksDBStateBackend it will be backed by disc. Nevertheless it is vital to allow clearing the s

Re: Flink CEP State Change Pattern

Hi Philip! I’m looping in Kostas to this thread. He might be able to provide some insights for your question. Cheers, Gordon On 14 October 2017 at 8:54:45 PM, Philip Limbeck (philiplimb...@gmail.com) wrote: Hi! I am quite new to Flink CEP and try to define a state change pattern with it.

Re: Flink CEP questions

Thank you very much Biplob and David Thanks David for those links . That is exactly what I was looking for. On Fri, Aug 18, 2017 at 5:16 AM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > Hi Basanth, > > Ad.3 Unfortunately right now, you cannot reset, but there is ongoing work > to i

Re: Flink CEP questions

Hi Basanth, Ad.3 Unfortunately right now, you cannot reset, but there is ongoing work to introduce AfterMatchSkipStrategies(https://issues.apache.org/jira/browse/FLINK-7169?filter=12339990). This will allow the behaviour you described with the SKIP_PAST_LAST strategy. Ad.4 If I understand corr

Re: Flink CEP questions

Hi Basanth, AFAIK, CEP works like sessions window and a session is started for each event which comes in and expires at the end of the time limit. Technically the count is kept separately for each event, so there's no reset. For ex, if you have 6 events, 1,2,3,4,5,6 (and they arrive in order lik

Re: Flink CEP questions

Hi Kostas, For 3 -> I was able to do the following and it worked perfectly fine. Is there a way we could reset? Looks like the following code behaves more like a sliding count. What I want to do is reset the count once the alert has matched, and start over the count. May be I will have to have som

Re: Flink CEP questions

Hi Basanth, This is the documentation page can be found here: https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/libs/cep.html For: 3) you should use the times(N) and the within(TIME) clauses

Re: Flink CEP issues

Hi Daiqing, Is it possible to share your job in order to reproduce the problem? Or at least a minimal example. If you see from the JIRA, there is another user in https://issues.apache.org/jira/browse/FLINK-6321 who had a similar problem but we

Re: Flink CEP not emitting timed out events properly

Hi Kostas, I ended up setting my currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp); to currentMaxTimestamp = Math.min(timestamp, currentMaxTimestamp); and changing this : if(firstEventFlag && (currentTime - systemTimeSinceLastModification > 1)){ systemTimeSinceLastModi

Re: Flink CEP not emitting timed out events properly

I know that there wouldn't be a scenario where the first event type(coming from topic t1) would be coming with a timestamp higher than the current watermark. Although I am still investigating whether the other events from other topics (specifically t3 and t4) are arriving after the watermark update

Re: Flink CEP not emitting timed out events properly

Are you sure that after incrementing the wm by 1sec, there is no element that will come with a timestamp smaller than this? Or, that after 10sec of inactivity, no element will come with such a timestamp? Kostas > On Jun 20, 2017, at 4:18 PM, Biplob Biswas wrote: > > currentMaxTimestamp = cur

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Yes, I have a flag in my timestampextractor. As you can see from the code below, I am checking whether currentTime - systemTimeSinceLastModification > 10 sec. as new events come then the watermark wouldn't be incremented. But as soon as I have a difference of more than 10 seconds,

Re: Flink CEP not emitting timed out events properly

You are correct that elements are waiting until a watermark with a higher timestamp than theirs (or the patterns timeout) arrives. Now for the Watermark emitter, 1) how do you measure the 10sec in processing time and ii) by how much do you advance the watermark. If you advance it by a lot, th

Re: Flink CEP not emitting timed out events properly

Hi Kostas, I have out-of-orderness of around 5 seconds from what I have observed but that too from events coming from a different topic. The initial topic doesn't have out-of-order events still I have added a generous time bound of 20 seconds. Still, I will try for a higher number just in order to

Re: Flink CEP not emitting timed out events properly

Hi Biplob, You are correct that only a higher watermark leads to discarded events. Are you sure that your custom watermark emitter does not emit a high watermark? E.g. your partition has elements that are far out-of-order. In addition, are you sure that your elements are not simply buffered and

Re: Flink CEP not emitting timed out events properly

But if that's the case, I don't understand why some of my events are just lost If the watermark which is used is the smallest ... They either I expect a match or I expect a timed out event. The only way I can imagine my events getting lost is higher watermark than the incoming event and thus

Re: Flink CEP not emitting timed out events properly

It is possible that two Watermarks will be passed through the Stream, but at an operator that has more than one input streams(in your case from different topics and partitions) the smallest Watermark is considered valid and passed downstream. Z pozdrowieniami! / Cheers! Dawid Wysakowicz *Data/So

Re: Flink CEP not emitting timed out events properly

Hi dawid, First of all congratulations on being a Flink committer, saw your tweet in the morning. Now regarding that link, that talks about multiple partitions for a single topic, here I am talking about multiple topics each having different number of partitions. I tried adding tinestampextract

Re: Flink CEP not emitting timed out events properly

Hi Biplop, If an operator has two inputs, the resulting Watermark is the smallest one from the two upstreams. More on that you can check here .

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Implementing my custom timestamp assigned made em realise a problem which we have in our architecture you may say. Any inputs would be really appreciated. So, for now, we are reading from 4 different kafka topics, and we have a flow similar to something like this: Event 1(on topic t

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Thanks for that suggestion, I would try that next, I have out of order events on one of my Kafka topics and that's why I am using BoundedOutOfOrdernessTimestampExtractor(), now that this doesn't work as expected I would try to work with the Base class as you suggested. Although this b

Re: Flink CEP not emitting timed out events properly

Hi Biplob, If you know what you want, you can always write your custom AssignerWithPeriodicWatermarks that does your job. If you want to just increase the watermark, you could simply check if you have received any elements and if not, emit a watermark with the timestamp of the previous watermark

Re: Flink CEP not emitting timed out events properly

Hi Kostas, Thanks for the reply, makes things a bit more clear. Also, I went through this link and it is something similar I am trying to observe. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Listening-to-timed-out-patterns-in-Flink-CEP-td9371.html I am checking for timed

Re: Flink CEP not emitting timed out events properly

Hi Biplob, With processing time there are no watermarks in the stream. The problem that you are seeing is because in processing time, the CEP library expects the “next” element to come, in order to investigate if some of the patterns have timed-out. Kostas > On Jun 16, 2017, at 1:29 PM, Biplob

Re: Flink CEP

Hi Daniel, The NOT operation is not yet supported in the CEP library but there is an open issue https://issues.apache.org/jira/browse/FLINK-3320 and we are working on integrating it in the next release of the CEP library. Please monitor the is

Re: Flink CEP

Hi Flink team, Any ideas on how to achieve the below outcome: I have a stream of Strings, with the words Start, Step and Stop. I'm trying to build a pattern for a Start followed by another Start, without a Stop in between. e.g. this would give an alert due to lines 5 and 8: Start Step Step Stop

Re: Flink CEP dynamic patterns

thank you, i will investigate further On Mon, Dec 5, 2016 at 10:36 AM, Till Rohrmann wrote: > Hi Abdallah, > > I've answered your question on SO. For the sake of completeness here is a > copy: > > At the moment Flink's CEP library does not support dynamic pattern changes > out of the box. Thus,

Re: Flink CEP dynamic patterns

Hi Abdallah, I've answered your question on SO. For the sake of completeness here is a copy: At the moment Flink's CEP library does not support dynamic pattern changes out of the box. Thus, once you've defined your pattern and started your job, it will only process this defined pattern. However,

Re: Flink : CEP processing

Hi, Sameet is right about the snapshotting. The CEP operator behaves more or less like a FlatMap operator that keeps some more complex state internally. Snapshotting works the same as with any other operator. Cheers, Aljoscha On Thu, 11 Aug 2016 at 00:54 Sameer W wrote: > Mans, > > I think at t

Re: Flink : CEP processing

Mans, I think at this time we need someone who knows the internal implementation to answer definitively- My understanding is- 1. Internally CEP is like a map operator with session-like semantics operating in a pipeline. You could do what it does but you would have to implement all that. If you n

Re: Flink : CEP processing

Thanks for the pointers Sameer. The reason I wanted to find out about snapshotting with CEP is because I thought that CEP state might also be snapshotted for recovery. If that is the case, then there are events in the CEP might be in two snapshots. Mans On Tuesday, August 9, 2016 1:15 PM,

Re: Flink : CEP processing

In one of the earlier thread Till explained this to me ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/CEP-and-Within-Clause-td8159.html ) 1. Within does not use time windows. It sort of uses session windows where the session begins when the first event of the pattern is iden

Re: Flink : CEP processing

Hi Sameer: If we use a within window for event series -  1. Does it interfere with the default time windows ?2. How does it affect snapshotting ?  3. If the window is too large are the events stored in a "processor" for the window to expire ?4. Are there any other know limitations and best pract

Re: Flink : CEP processing

In that case you need to get them into one stream somehow (keyBy a dummy value for example). There is always some logical key to keyBy on when data is arriving from multiple sources (ex some portion of the time stamp). You are looking for patterns within something (events happening around the s

Re: Flink : CEP processing

Thanks Sameer. So does that mean that if the events keys are not same we cannot use the CEP pattern match ?  What if events are coming from different sources and need to be correlated ? Mans On Tuesday, August 9, 2016 9:40 AM, Sameer W wrote: Hi, You will need to use keyBy operation fi

Re: Flink : CEP processing

Hi, You will need to use keyBy operation first to get all the events you need monitored in a pattern on the same node. Only then can you apply Pattern because it depends on the order of the events (first, next, followed by). I even had to make sure that the events were correctly sorted by timestam

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi Norman, sorry for the late reply. I finally found time and could, thanks to you, reproduce the problem. The problem was that the window borders were treated differently in two parts of the code. Now the left border of a window is inclusive and the right border (late elements) is exclusive. I've

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi, here is an example input that produces the error. These are read from Kafka. 01:43:43.5921 2121{"Pressure target - Value":"6"} 01:43:43.5961 2121{"Flow target - Value":"23"} 01:43:44.2631 2121{"Pressure target - Value":"7"} 01:43:44.9721 2121

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi Norman, could you provide me an example input data set which produces the error? E.g. the list of strings you inserted into Kafka/read from Kafka? Cheers, Till On Thu, Apr 7, 2016 at 11:05 AM, norman sp wrote: > Hi Till, > thank you. here's the code: > > public class CepStorzSimulator { > >

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi Till, thank you. here's the code: public class CepStorzSimulator { public static void main(String[] args) throws Exception { final ParameterTool parameterTool = ParameterTool.fromArgs(args); if(parameter

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi Norman, this error is exactly what I thought I had fixed. I guess there is still another case where a premature pruning can happen in the SharedBuffer. Could you maybe send me the example code with which you could produce the error. The input data would also be very helpful. Then I can debug it

Re: Flink CEP AbstractCEPPatternOperator fail after event detection

Hi Till, I used Flink version 1.0.0 and tried all three TimeCharacteristics. Not I tried the new Flink 1.0.1 that gives me the following error. After detecting an event it processes a few stream tuples but then crashes. I'm not sure how to solve that part of the error message: "This can indicate t

  1   2   >