First of all, it's a great design document. Looking forward having stream SQL in the foreseeable future :-)
I think it is a good idea to consolidate stream SQL and CEP in the long run. CEP's additional features compared to SQL boil down to pattern detection. Once we have this, it should be only a question of defining the SQL syntax for event patterns in order to integrate CEP with stream SQL. Oracle has already defined an extension [1] to detect patterns in a set of table rows. This or Esper's event processing language (EPL) [2] could be a good starting point. [1] https://docs.oracle.com/database/121/DWHSG/pattern.htm#DWHSG8959 [2] http://www.espertech.com/esper/release-5.2.0/esper-reference/html/ Cheers, Till On Mon, Jan 11, 2016 at 10:12 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Thanks for the feedback! > > We will start the SQL effort with putting the existing (batch) Table API on > top of Apache Calcite. > From there we continue to add streaming support for the Table API before we > put a StreamSQL interface on top. > > Consolidating the efforts with the CEP library sounds like a good idea to > me. > Maybe it can be nicely integrated with the streaming table API and later as > well with the StreamSQL interface (the StreamSQL dialect is not defined > yet). > > @Till: What do you think about adding CEP features to the Table API. From > the CEP design doc, it looks like we need to add a pattern matching > operator in addition to the window features that we need to add for > streaming Table API in any case. > > Best, Fabian > > 2016-01-11 4:03 GMT+01:00 Jiangsong (Hi) <hi.jiangs...@huawei.com>: > > > I suggest refering to Esper EPL[1], which is a SQL-standard language > > extend to offering a cluster of window, pattern matching. EPL can both > > support Streaming SQL and CEP with one unified syntax. > > > > [1] > > > http://www.espertech.com/esper/release-5.2.0/esper-reference/pdf/esper_reference.pdf > > (Chapter 5. EPL Reference: Clauses) > > > > > > Regards > > Song > > > > > > -----邮件原件----- > > 发件人: Chiwan Park [mailto:chiwanp...@apache.org] > > 发送时间: 2016年1月11日 10:31 > > 收件人: dev@flink.apache.org > > 主题: Re: Effort to add SQL / StreamSQL to Flink > > > > We still don’t have a concensus about the streaming SQL and CEP library > on > > Flink. Some people want to merge these two libraries. Maybe we have to > > discuss about this in mailing list. > > > > > On Jan 11, 2016, at 10:53 AM, Nick Dimiduk <ndimi...@gmail.com> wrote: > > > > > > What's the relationship between the streaming SQL proposed here and > > > the CEP syntax proposed earlier in the week? > > > > > > On Sunday, January 10, 2016, Henry Saputra <henry.sapu...@gmail.com> > > wrote: > > > > > >> Awesome! Thanks for the reply, Fabian. > > >> > > >> - Henry > > >> > > >> On Sunday, January 10, 2016, Fabian Hueske <fhue...@gmail.com > > >> <javascript:;>> wrote: > > >> > > >>> Hi Henry, > > >>> > > >>> There is https://issues.apache.org/jira/browse/FLINK-2099 and a few > > >>> subissues. > > >>> I'll reorganize these and add more issues for the tasks described in > > >>> the design document in the next days. > > >>> > > >>> Thanks, Fabian > > >>> > > >>> 2016-01-10 2:45 GMT+01:00 Henry Saputra <henry.sapu...@gmail.com > > >> <javascript:;> > > >>> <javascript:;>>: > > >>> > > >>>> HI Fabian, > > >>>> > > >>>> Have you created JIRA ticket to keep track of this new feature? > > >>>> > > >>>> - Henry > > >>>> > > >>>> On Thu, Jan 7, 2016 at 6:05 AM, Fabian Hueske <fhue...@gmail.com > > >> <javascript:;> > > >>> <javascript:;>> wrote: > > >>>>> Hi everybody, > > >>>>> > > >>>>> in the last days, Timo and I refined the design document for > > >>>>> adding a > > >>>> SQL / > > >>>>> StreamSQL interface on top of Flink that was started by Stephan. > > >>>>> > > >>>>> The document proposes an architecture that is centered around > > >>>>> Apache Calcite. Calcite is an Apache top-level project and > > >>>>> includes a SQL > > >>>> parser, > > >>>>> a semantic validator for relational queries, and a rule- and > > >> cost-based > > >>>>> relational optimizer. Calcite is used by Apache Hive and Apache > > >>>>> Drill (among other projects). In a nutshell, the plan is to > > >>>>> translate Table > > >>> API > > >>>>> and SQL queries into Calcite's relational expression trees, > > >>>>> optimize > > >>>> these > > >>>>> trees, and translate them into DataSet and DataStream programs.The > > >>>> document > > >>>>> breaks down the work into several tasks and subtasks. > > >>>>> > > >>>>> Please review the design document and comment. > > >>>>> > > >>>>> -- > > > >>>>> > > >>>> > > >>> > > >> https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjP > > >> cp1h2TVqdI/edit?usp=sharing > > >>>>> > > >>>>> Unless there are major concerns with the design, Timo and I want > > >>>>> to > > >>> start > > >>>>> next week to move the current Table API on top of Apache Calcite > > >> (Task > > >>> 1 > > >>>> in > > >>>>> the document). The goal of this task is to have the same > > >> functionality > > >>> as > > >>>>> currently, but with Calcite in the translation process. This is a > > >>>> blocking > > >>>>> task that we hope to complete soon. Afterwards, we can > > >>>>> independently > > >>> work > > >>>>> on different aspects such as extending the Table API, adding a SQL > > >>>>> interface (basically just a parser), integration with external > > >>>>> data sources, better code generation, optimization rules, > > >>>>> streaming > > >> support > > >>>> for > > >>>>> the Table API, StreamSQL, etc.. > > >>>>> > > >>>>> Timo and I plan to work on a WIP branch to implement Task 1 and > > >>>>> merge > > >>> it > > >>>> to > > >>>>> the master branch once the task is completed. Of course, everybody > > >>>>> is welcome to contribute to this effort. Please let us know such > > >>>>> that we > > >>> can > > >>>>> coordinate our efforts. > > >>>>> > > >>>>> Thanks, > > >>>>> Fabian > > > > Regards, > > Chiwan Park > > > > > > >
