Thanks Leonard for the clarifications! On Mon, Aug 17, 2020 at 9:17 PM Leonard Xu <xbjt...@gmail.com> wrote:
> > > But are we still able to track different views of such a > > table through time, as rows are added/deleted to/from the table? > > Yes, in fact we support temporal table from changlog which contains all > possible message types(INSERT/UPDATE/DELETE). > > > For > > example, suppose I have an append-only table source with event-time and > PK, > > will I be allowed to do an event-time temporal join with this table? > Yes, I list some examples in the doc, the example versioned_rates3 is > this case exactly. > > Best > Leonard > > > > > > On Wed, Aug 12, 2020 at 3:31 PM Leonard Xu <xbjt...@gmail.com <mailto: > xbjt...@gmail.com>> wrote: > > > >> Hi, all > >> > >> After a detailed offline discussion about the temporal table related > >> concept and behavior, we had a reliable solution and rejected several > >> alternatives. > >> Compared to rejected alternatives, the proposed approach is a more > unified > >> story and also friendly to user and current Flink framework. > >> I improved the FLIP[1] with the proposed approach and refactored the > >> document organization to make it clear enough. > >> > >> Please let me know if you have any concerns, I’m looking forward your > >> comments. > >> > >> > >> Best > >> Leonard > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > < > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > > > >>> > >> > >> > >>> 在 2020年8月4日，21:25，Leonard Xu <xbjt...@gmail.com <mailto: > xbjt...@gmail.com>> 写道： > >>> > >>> Hi, all > >>> > >>> I’ve updated the FLIP[1] with the terminology `ChangelogTime`. > >>> > >>> Best > >>> Leonard > >>> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > < > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > > > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > < > https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL > > > >>> > >>> > >>>> 在 2020年8月4日，20:58，Leonard Xu <xbjt...@gmail.com <mailto: > xbjt...@gmail.com> <mailto: > >> xbjt...@gmail.com <mailto:xbjt...@gmail.com>>> 写道： > >>>> > >>>> Hi, Timo > >>>> > >>>> Thanks for you response. > >>>> > >>>>> 1) Naming: Is operation time a good term for this concept? If I read > >> "The operation time is the time when the changes happened in system." or > >> "The system time of DML execution in database", why don't we call it > >> `ChangelogTime` or `SystemTime`? Introducing another terminology of > time in > >> Flink should be thought through. > >>>> > >>>> I agree that we should thought through. I have considered the name > >> `ChangelogTime` and `SystemTime` too, I don’t have strong opinion on the > >> name. > >>>> > >>>> I proposed `operationTime` because most changelog comes from Database > >> and we always called an action as `operation` rather than `change` in > >> Database, the operation time is easier to understand for database > users, > >> but it's more like a database terminology. > >>>> > >>>> For `SystemTime`, user may confuse which one does the system in > >> `SystemTime` represents? Flink, Database or CDC tool. Maybe it’s not a > >> good name. > >>>> > >>>> `ChangelogTime` is a pretty choice which is more unified with existed > >> terminology `Changelog` and `ChangelogMode`, so let me use > `ChangelogTime` > >> and I’ll update the FLIP. > >>>> > >>>> > >>>>> 2) Exposing it through `org.apache.flink.types.Row`: Shall we also > >> expose the concept of time through the user-level `Row` type? The FLIP > does > >> not mention this explictly. I think we can keep it as an internal > concept > >> but I just wanted to ask for clarification. > >>>> > >>>> Yes, I want to keep it as an internal concept, we have discussed that > >> changelog time concept should be the third time concept(the other two > are > >> event-time and processing-time). It’s not easy for normal users(or to > help > >> normal users) understand the three concepts accurately, and I did not > find > >> a big enough scenario that user need to touch the changelog time for > now, > >> so I tend to do not expose the concept to users. > >>>> > >>>> > >>>> Best, > >>>> Leonard > >>>> > >>>> > >>>>> > >>>>> On 04.08.20 04:58, Leonard Xu wrote: > >>>>>> Thanks Konstantin, > >>>>>> Regarding your questions, hope my comments has address your > questions > >> and I also add a few explanation in the FLIP. > >>>>>> Thank you all for the feedback, > >>>>>> It seems everyone involved in this thread has reached a consensus. > >>>>>> I will start a vote thread later. > >>>>>> Best, > >>>>>> Leonard > >>>>>>> 在 2020年8月3日，19:35，godfrey he <godfre...@gmail.com <mailto: > godfre...@gmail.com> <mailto: > >> godfre...@gmail.com <mailto:godfre...@gmail.com>>> 写道： > >>>>>>> > >>>>>>> Thanks Lennard for driving this FLIP. > >>>>>>> Looks good to me. > >>>>>>> > >>>>>>> Best, > >>>>>>> Godfrey > >>>>>>> > >>>>>>> Jark Wu <imj...@gmail.com <mailto:imj...@gmail.com> <mailto: > imj...@gmail.com <mailto:imj...@gmail.com>>> 于2020年8月3日周一 > >> 下午12:04写道： > >>>>>>> > >>>>>>>> Thanks Leonard for the great FLIP. I think it is in very good > shape. > >>>>>>>> +1 to start a vote. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Jark > >>>>>>>> > >>>>>>>> On Fri, 31 Jul 2020 at 17:56, Fabian Hueske <fhue...@gmail.com > <mailto:fhue...@gmail.com> > >> <mailto:fhue...@gmail.com <mailto:fhue...@gmail.com>>> wrote: > >>>>>>>> > >>>>>>>>> Hi Leonard, > >>>>>>>>> > >>>>>>>>> Thanks for this FLIP! > >>>>>>>>> Looks good from my side. > >>>>>>>>> > >>>>>>>>> Cheers, Fabian > >>>>>>>>> > >>>>>>>>> Am Do., 30. Juli 2020 um 22:15 Uhr schrieb Seth Wiesman < > >>>>>>>>> sjwies...@gmail.com <mailto:sjwies...@gmail.com> <mailto: > sjwies...@gmail.com <mailto:sjwies...@gmail.com>> > >>>>>>>>>> : > >>>>>>>>> > >>>>>>>>>> Hi Leondard, > >>>>>>>>>> > >>>>>>>>>> Thank you for pushing this, I think the updated syntax looks > >> really > >>>>>>>> good > >>>>>>>>>> and the semantics make sense to me. > >>>>>>>>>> > >>>>>>>>>> +1 > >>>>>>>>>> > >>>>>>>>>> Seth > >>>>>>>>>> > >>>>>>>>>> On Wed, Jul 29, 2020 at 11:36 AM Leonard Xu <xbjt...@gmail.com > <mailto:xbjt...@gmail.com> > >> <mailto:xbjt...@gmail.com <mailto:xbjt...@gmail.com>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, Konstantin > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 1) A "Versioned Temporal Table DDL on source" can only be > >> joined > >>>>>>>> on > >>>>>>>>>> the > >>>>>>>>>>>> PRIMARY KEY attribute, correct? > >>>>>>>>>>> Yes, the PRIMARY KEY would be join key. > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 2) Isn't it the time attribute in the ORDER BY clause of the > >> VIEW > >>>>>>>>>>> definition that defines > >>>>>>>>>>>> whether a event-time or processing time temporal table join is > >>>>>>>> used? > >>>>>>>>>>> > >>>>>>>>>>> I think event-time or processing-time temporal table join > >> depends on > >>>>>>>>> fact > >>>>>>>>>>> table’s time attribute in temporal join rather than from > temporal > >>>>>>>> table > >>>>>>>>>>> side, the event-time or processing time in temporal table is > just > >>>>>>>> used > >>>>>>>>> to > >>>>>>>>>>> split the validity period of versioned snapshot of temporal > >> table. > >>>>>>>> The > >>>>>>>>>>> processing time attribute is not necessary for temporal table > >>>>>>>> without > >>>>>>>>>>> version, only the primary key is required, the following VIEW > is > >> also > >>>>>>>>>> valid > >>>>>>>>>>> for temporal table without version. > >>>>>>>>>>> CREATE VIEW latest_rates AS > >>>>>>>>>>> SELECT currency, LAST_VALUE(rate) -- only keep the > >> latest > >>>>>>>>>>> version > >>>>>>>>>>> FROM rates > >>>>>>>>>>> GROUP BY currency; -- inferred > primary > >> key > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 3) A "Versioned Temporal Table DDL on source" is always > >> versioned > >>>>>>>> on > >>>>>>>>>>>> operation_time regardless of the lookup table attribute > >> (event-time > >>>>>>>>> or > >>>>>>>>>>>> processing time attribute), correct? > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Yes, the semantics of `FOR SYSTEM_TIME AS OF o.time` is using > the > >>>>>>>>> o.time > >>>>>>>>>>> value to lookup the version of the temporal table. > >>>>>>>>>>> For fact table has the processing time attribute, it means only > >>>>>>>> lookup > >>>>>>>>>> the > >>>>>>>>>>> latest version of temporal table and we can do some > optimization > >> in > >>>>>>>>>>> implementation like only keep the latest version. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Best > >>>>>>>>>>> Leonard > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >>> > >> > >> > > > > -- > > Best regards! > > Rui Li > > -- Best regards! Rui Li
