Hi Jark, Thanks for the summary! I like the proposal!
It makes it very clear that an event time attribute is an existing column on which watermark metadata is defined whereas a processing time attribute is a computed field. I have one comment regarding the section on "Complex Watermark Strategies". The proposal says that you can also use a scalar function. I don't think that a "text book" scalar function would be sufficient for more advanced strategies. For example a histogram-based approach would need to remember the values of the last x records. The interface of a scalar function would still work for that, but it would be a stateful function (which would not be OK for a scalar function). I don't think it's a problem, but wanted to mention it here. Best, Fabian Am Do., 19. Sept. 2019 um 18:05 Uhr schrieb Jark Wu <imj...@gmail.com>: > Hi everyone, > > Thanks all for the valuable suggestions and feedbacks so far. > Before starting the vote, I would like to summarize the proposed DDL syntax > in the mailing list. > > ## Rowtime Attribute (Watermark Syntax) > > CREATE TABLE table_name ( > WATERMARK FOR <columnName> AS <watermark_strategy_expression> > ) WITH ( > ... > ) > > It marks an existing field <columnName> as the rowtime attribute, and the > watermark is generated by the expression <watermark_strategy_expression>. > <watermark_strategy_expression> can be arbitrary expression which returns a > nullable BIGINT or TIMESTAMP as the watermark value. > > For common cases, users can use the following expressions to define a > strategy. > 1. Bounded Out of Orderness, the strategy can be "rowtimeField - INTERVAL > 'string' timeUnit". > 2. Preserve Watermark From Source, the strategy can be > "SYSTEM_WATERMARK()". > > ## Proctime Attribute > > CREATE TABLE table_name ( > ... > proc AS SYSTEM_PROCTIME() > ) WITH ( > ... > ) > > It uses the computed column syntax to add an additional column with > proctime attribute. Here SYSTEM_PROCTIME() is a built-in function. > > For more details and the implementations, please refer to the design doc: > > https://docs.google.com/document/d/1-SecocBqzUh7zY6HBYcfMlG_0z-JAcuZkCvsmN3LrOw/edit?ts=5d822dba > > Feel free to leave your further feedbacks! > > Thanks, > Jark > > On Thu, 19 Sep 2019 at 11:23, Kurt Young <ykt...@gmail.com> wrote: > > > +1 to start vote process. > > > > Best, > > Kurt > > > > > > On Thu, Sep 19, 2019 at 10:54 AM Jark Wu <imj...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > Thanks all for joining the discussion in the doc[1]. > > > It seems that the discussion is converged and there is a consensus on > the > > > current FLIP document. > > > If there is no objection, I would like to convert it into cwiki FLIP > page > > > and start voting process. > > > > > > For more details, please refer to the design doc (it is slightly > changed > > > since the initial proposal). > > > > > > Thanks, > > > Jark > > > > > > [1]: > > > > > > > > > https://docs.google.com/document/d/1-SecocBqzUh7zY6HBYcfMlG_0z-JAcuZkCvsmN3LrOw/edit?ts=5d8258cd > > > > > > On Mon, 16 Sep 2019 at 16:12, Kurt Young <ykt...@gmail.com> wrote: > > > > > > > After some review and discussion in the google document, I think it's > > > time > > > > to > > > > convert this design to a cwiki flip page and start voting process. > > > > > > > > Best, > > > > Kurt > > > > > > > > > > > > On Mon, Sep 9, 2019 at 7:46 PM Jark Wu <imj...@gmail.com> wrote: > > > > > > > > > Hi all, > > > > > > > > > > Thanks all for so much feedbacks received in the doc so far. > > > > > I saw a general agreement on using computed column to support > > proctime > > > > > attribute and extract timestamps. > > > > > So we will prepare a computed column FLIP and share in the dev ML > > soon. > > > > > > > > > > Feel free to leave more comments! > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > > > > > > On Fri, 6 Sep 2019 at 13:50, Dian Fu <dian0511...@gmail.com> > wrote: > > > > > > > > > > > Hi Jark, > > > > > > > > > > > > Thanks for bringing up this discussion and the detailed design > doc. > > > > This > > > > > > is definitely a critical feature for streaming SQL jobs. I have > > left > > > a > > > > > few > > > > > > comments in the design doc. > > > > > > > > > > > > Thanks, > > > > > > Dian > > > > > > > > > > > > > 在 2019年9月6日，上午11:48，Forward Xu <forwardxu...@gmail.com> 写道： > > > > > > > > > > > > > > Thanks Jark for this topic, This will be very useful. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > ForwardXu > > > > > > > > > > > > > > Danny Chan <yuzhao....@gmail.com> 于2019年9月6日周五 上午11:26写道： > > > > > > > > > > > > > >> Thanks Jark for bring up this topic, this is definitely an > > import > > > > > > feature > > > > > > >> for the SQL, especially the DDL users. > > > > > > >> > > > > > > >> I would spend some time to review this design doc, really > > thanks. > > > > > > >> > > > > > > >> Best, > > > > > > >> Danny Chan > > > > > > >> 在 2019年9月6日 +0800 AM11:19，Jark Wu <imj...@gmail.com>，写道： > > > > > > >>> Hi everyone, > > > > > > >>> > > > > > > >>> I would like to start discussion about how to support time > > > > attribute > > > > > in > > > > > > >> SQL > > > > > > >>> DDL. > > > > > > >>> In Flink 1.9, we already introduced a basic SQL DDL to > create a > > > > > table. > > > > > > >>> However, it doesn't support to define time attributes. This > > makes > > > > > users > > > > > > >>> can't > > > > > > >>> apply window operations on the tables created by DDL which > is a > > > bad > > > > > > >>> experience. > > > > > > >>> > > > > > > >>> In FLIP-66, we propose a syntax for watermark to define > rowtime > > > > > > attribute > > > > > > >>> and propose to use computed column syntax to define proctime > > > > > attribute. > > > > > > >>> But computed column is another big topic and should deserve a > > > > > separate > > > > > > >>> FLIP. > > > > > > >>> If we have a consensus on the computed column approach, we > will > > > > start > > > > > > >>> computed column FLIP soon. > > > > > > >>> > > > > > > >>> FLIP-66: > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1-SecocBqzUh7zY6HBYcfMlG_0z-JAcuZkCvsmN3LrOw/edit# > > > > > > >>> > > > > > > >>> Thanks for any feedback! > > > > > > >>> > > > > > > >>> Best, > > > > > > >>> Jark > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
