Hi all, Integrating SQL with CEP seems like a really nice idea. Unfortunately I had time just for a brief look at the design doc, but it looks really cool and thorough. Also will have a second run tomorrow and will try to provide more comments. Anyway will be glad to help pushing the initiative forward.
Z pozdrowieniami! / Cheers! Dawid Wysakowicz *Data/Software Engineer* Skype: dawid_wys | Twitter: @OneMoreCoder <http://getindata.com/> 2017-06-13 8:19 GMT+02:00 yueting chen <yestinm...@gmail.com>: > Hi Dian & Jark, > > I checked out your prototype code, but it didn't pass the CEPITCase test in > the flink-table component. > It turns out that in the `MatchCodeGenerator.scala` file, line 74 should > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}` instead > of > `${classOf[IterativeCondition.Context[_]]}`. > > I've also read your desgin document and it looks fine to me. Actually, I am > working on the same thing recently, I think maybe we can work together to > push this forward. > > Thanks, > Yueting Chen > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <dia...@apache.org> wrote: > > > Hi Fabian, > > > > We have evaluated the missing features of Flink CEP roughly, it should > not > > be quite difficult to support them. Kostas, Dawid, what's your thought? > > > > For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs > and > > start to work right now or we should wait until the release of calcite > > 1.13? > > > > Btw, could you help to add me(dian.fu) to the contributor list, then I > can > > assign the JIRAs to myself? Thanks a lot. > > > > Best regards, > > Dian > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <fhue...@gmail.com> > wrote: > > > > > Hi Jark, > > > > > > Thanks for updating the design doc and sharing your prototype! > > > I didn't look at the code in detail, but the fact that it is less than > 1k > > > LOC is very promising. It seems that most of the complex CEP logic can > be > > > reused :-) > > > Adding a dependency on flink-cep should be fine, IMO. It is a very slim > > > library with almost none external dependencies. > > > > > > Regarding the missing features of Flink CEP that you listed in the > design > > > doc, it would be good to get some in put from Kostas and Dawid which > are > > > the main contributors to CEP. > > > Do you have already plans regarding some of the missing features or can > > you > > > assess how hard it would be to integrate them? > > > > > > Cheers, Fabian > > > > > > Btw. The Calcite community started a discussion about releasing Calcite > > > 1.13. So, the missing features might soon be available. > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <j...@apache.org>: > > > > > > > Hi guys, > > > > > > > > Good news! We have made a prototype for integrating CEP and SQL. See > > this > > > > link > > > > https://github.com/apache/flink/compare/master... > > > > wuchong:cep-on-sql?expand=1 > > > > > > > > > > > > You can check CepITCase to try the simple CQL example. > > > > > > > > Meanwhile, we updated our design doc with additional implementation > > > detail, > > > > including how > > > > to translate MATCH_RECOGNIZE into CEP API, and the features needed to > > add > > > > to Flink CEP, > > > > and the implementation plan. See the document > > > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3 > > > > > > > > In the prototype, we make flink-table dependency on flink-cep. Do you > > > think > > > > that is fine? > > > > > > > > What do you think about the prototype and the design doc ? > > > > > > > > Any feedbacks are welcome! > > > > > > > > Cheers, > > > > Jark Wu > > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <trohrm...@apache.org>: > > > > > > > > > Thanks for sharing your ideas with the community. I really like the > > > > design > > > > > document and think that it's a good approach to follow Oracle's SQL > > > > > extension for pattern matching. Looking forward to having support > for > > > SQL > > > > > with CEP capabilities :-) > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <j...@apache.org> wrote: > > > > > > > > > > > Hi @Kostas, @Fabian, thank you for your support. > > > > > > > > > > > > @Fabian, I totally agree with you that we should focus on SQL > > first. > > > > > Let's > > > > > > keep Table API in mind and discuss that later. > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure about that. I > think > > > it > > > > > > makes sense to make it required in streaming mode(either order by > > > > rowtime > > > > > > or order by proctime). But CEP also works in batch mode, and not > > > > > necessary > > > > > > to order by some column. Nevertheless, we can support CEP on > batch > > > SQL > > > > > > later. > > > > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP > library > > > > (with > > > > > > NFA, CEP operator). And we will output a detailed doc and a > > prototype > > > > in > > > > > > the next days. > > > > > > > > > > > > Regards, > > > > > > Jark Wu > > > > > > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <fhue...@gmail.com>: > > > > > > > > > > > >> Thanks Dian and Jark for this proposal! > > > > > >> > > > > > >> As you wrote, Till and I (and Kostas) have been thinking about > > this > > > > for > > > > > >> some time but haven't had time to work on this feature. > > > > > >> I think it would be a great addition and value add for Flink's > SQL > > > > > >> support and Table API. > > > > > >> > > > > > >> I read the proposal and think it is very good. We might need to > > add > > > a > > > > > bit > > > > > >> more details, esp. when planning the concrete steps of the > > > > > implementation. > > > > > >> > > > > > >> A few comments to the proposal: > > > > > >> - IMO, the development should start focusing on SQL and its > > > semantics. > > > > > >> Pattern support for the Table API should be added later. We > > followed > > > > > that > > > > > >> approach for the OVER windows and I think it worked quiet well. > > > > > >> - We probably want to reuse as much as possible from the CEP > > > library. > > > > > >> That means we need to check if the semantics of the CEP library > > and > > > > > >> Oracle's PATTERN syntax are aligned (or how we can express the > > > PATTERN > > > > > >> semantics with the CEP library). This should be one of the first > > > > steps, > > > > > IMO. > > > > > >> - I would make the orderBy() clause required. In regular SQL > rows > > > have > > > > > no > > > > > >> order, so we need to make that explicit (this would also be > > > consistent > > > > > with > > > > > >> the OVER windows). > > > > > >> > > > > > >> Let me know what you think. > > > > > >> > > > > > >> Best, Fabian > > > > > >> > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas < > > > > k.klou...@data-artisans.com > > > > > >: > > > > > >> > > > > > >>> Thanks a lot for opening the discussion! > > > > > >>> > > > > > >>> This is a really interesting idea that has been in our heads > > > > > >>> since the first implementation of the CEP library. > > > > > >>> > > > > > >>> A big +1 for moving forward with this. > > > > > >>> > > > > > >>> And as for the design document, I will definitely have a look > > > > > >>> and comment there. > > > > > >>> > > > > > >>> Kostas > > > > > >>> > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <j...@apache.org> wrote: > > > > > >>> > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas > > > > > >>> > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <j...@apache.org>: > > > > > >>> > > > > > >>>> Hi devs, > > > > > >>>> > > > > > >>>> Dian and me and our teammates have investigated this for a > long > > > > time. > > > > > >>>> We think consolidating Flink SQL and CEP is an exciting thing > > for > > > > > Flink. > > > > > >>>> It'll make SQL more powerful and give users the ability to > > easily > > > > and > > > > > >>>> quickly build CEP applications. And I find Flink community > has > > > also > > > > > talked > > > > > >>>> about this idea before, such as the mailing list [1] and [2] > and > > > > > Fabian & > > > > > >>>> Till's talk in Flink Forward 2016 [3]. > > > > > >>>> > > > > > >>>> I think THIS IS THE POINT to bring up this topic again. > Because > > we > > > > > >>>> already have pattern matching foundation in Flink CEP library, > > and > > > > > Stream > > > > > >>>> SQL is ready now and Calcite has partially supported pattern > > > > matching > > > > > >>>> syntax! We also drafted a design doc about how to integrate > SQL > > > and > > > > > CEP, > > > > > >>>> and how to support CEP on Table API. > > > https://docs.google.com/docume > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp= > > > sharing > > > > > >>>> > > > > > >>>> > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this > > discussion, > > > > it > > > > > >>>> would be great to hear your response. > > > > > >>>> > > > > > >>>> > > > > > >>>> What do others think? > > > > > >>>> > > > > > >>>> > > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3. > > nabble.c > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787 > > > > > >>>> > > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3. > > nabble.c > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 > > > > > >>>> > > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming- > > analytics- > > > > > >>>> cep-two-sides-of-the-same-coin > > > > > >>>> > > > > > >>>> > > > > > >>>> Regards, > > > > > >>>> > > > > > >>>> Jark Wu > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <dia...@apache.org>: > > > > > >>>> > > > > > >>>>> Hi everyone, > > > > > >>>>> > > > > > >>>>> Flink's CEP library is a great library for complex event > > > > processing, > > > > > >>>>> more > > > > > >>>>> and more customers are expressing their interests in it. But > it > > > > also > > > > > >>>>> has > > > > > >>>>> some limitations that users usually have to write a lot of > code > > > > even > > > > > >>>>> for a > > > > > >>>>> very simple pattern match use case as it currently only > > supports > > > > the > > > > > >>>>> Java > > > > > >>>>> API. > > > > > >>>>> We have investigated some popular CEP products such as esper > > [1] > > > > and > > > > > >>>>> siddhi > > > > > >>>>> [2] and found that most of these CEP products support > SQL-like > > > > > >>>>> expressions > > > > > >>>>> such as EPL to describe the match pattern. But these > solutions > > > also > > > > > >>>>> have > > > > > >>>>> the drawbacks that the pattern match languages are not > standard > > > > SQL, > > > > > >>>>> the > > > > > >>>>> learn curve is steep for users and it's impossible to > integrate > > > > them > > > > > >>>>> into > > > > > >>>>> the Flink Table API & SQL. > > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new > > pattern > > > > > >>>>> recognition clause match_recognize which is a pattern > > recognition > > > > > >>>>> clause > > > > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes > > to > > > > > define > > > > > >>>>> match pattern in sql expression. Calcite already supports > part > > of > > > > > this > > > > > >>>>> standard [5]. I think it will be of great value to support > > > > > expressing > > > > > >>>>> pattern recognition clause with match_recognize clause by > > > > integrating > > > > > >>>>> it > > > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any > > > thoughts? > > > > > >>>>> > > > > > >>>>> [1] http://www.espertech.com > > > > > >>>>> [2] https://github.com/wso2/siddhi > > > > > >>>>> [3] > > > > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql- > > > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm# > > > CQLLR1531 > > > > > >>>>> [4] > > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat > > > > > >>>>> tern-recogniton-11.pdf > > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570 > > > > > >>>>> > > > > > >>>>> Best Regards, > > > > > >>>>> Dian > > > > > >>>>> > > > > > >>>> > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
