Thanks for the updated FLIP Leonard! In my opinion this was an improvement. So +1 for this design.
I have just one remark regarding the terminology. I find the term "Temporal Table without Version" somewhat confusing. IMO, versions are the core principle of temporal tables and temporal tables without versions don't make much sense to me. What makes such a table a "Temporal" table? Isn't it just a regular table? If I understand the proposal correctly, "Temporal Tables without Version" can only be used in processing time temporal table joins, because this join only requests the current version. But all regular tables can be used in processing time (temporal) table joins as well. It's basically the same as a lookup join, with the only difference that the table is maintained in Flink and not accessed in an external system (for example via JDBC). Are "Temporal Tables without Version" called "Temporal" because they can be used in "processing time temporal table joins" and due to its name this join needs to join something that's called "Temporal"? In that case, we might want to rename "processing time temporal table joins" into something else that does not imply a versioning. Maybe we can call them just lookup joins to avoid introducing another term? Thanks, Fabian Am Di., 18. Aug. 2020 um 04:30 Uhr schrieb Rui Li <lirui.fu...@gmail.com>: > Thanks Leonard for the clarifications! > > On Mon, Aug 17, 2020 at 9:17 PM Leonard Xu <xbjt...@gmail.com> wrote: > >> >> > But are we still able to track different views of such a >> > table through time, as rows are added/deleted to/from the table? >> >> Yes, in fact we support temporal table from changlog which contains all >> possible message types(INSERT/UPDATE/DELETE). >> >> > For >> > example, suppose I have an append-only table source with event-time and >> PK, >> > will I be allowed to do an event-time temporal join with this table? >> Yes, I list some examples in the doc, the example versioned_rates3 is >> this case exactly. >> >> Best >> Leonard >> >> >> > >> > On Wed, Aug 12, 2020 at 3:31 PM Leonard Xu <xbjt...@gmail.com <mailto: >> xbjt...@gmail.com>> wrote: >> > >> >> Hi, all >> >> >> >> After a detailed offline discussion about the temporal table related >> >> concept and behavior, we had a reliable solution and rejected several >> >> alternatives. >> >> Compared to rejected alternatives, the proposed approach is a more >> unified >> >> story and also friendly to user and current Flink framework. >> >> I improved the FLIP[1] with the proposed approach and refactored the >> >> document organization to make it clear enough. >> >> >> >> Please let me know if you have any concerns, I’m looking forward your >> >> comments. >> >> >> >> >> >> Best >> >> Leonard >> >> >> >> [1] >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> >> < >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> < >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> > >> >>> >> >> >> >> >> >>> 在 2020年8月4日，21:25，Leonard Xu <xbjt...@gmail.com <mailto: >> xbjt...@gmail.com>> 写道： >> >>> >> >>> Hi, all >> >>> >> >>> I’ve updated the FLIP[1] with the terminology `ChangelogTime`. >> >>> >> >>> Best >> >>> Leonard >> >>> [1] >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> < >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> > >> >> < >> >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> < >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-132+Temporal+Table+DDL >> > >> >>> >> >>> >> >>>> 在 2020年8月4日，20:58，Leonard Xu <xbjt...@gmail.com <mailto: >> xbjt...@gmail.com> <mailto: >> >> xbjt...@gmail.com <mailto:xbjt...@gmail.com>>> 写道： >> >>>> >> >>>> Hi, Timo >> >>>> >> >>>> Thanks for you response. >> >>>> >> >>>>> 1) Naming: Is operation time a good term for this concept? If I read >> >> "The operation time is the time when the changes happened in system." >> or >> >> "The system time of DML execution in database", why don't we call it >> >> `ChangelogTime` or `SystemTime`? Introducing another terminology of >> time in >> >> Flink should be thought through. >> >>>> >> >>>> I agree that we should thought through. I have considered the name >> >> `ChangelogTime` and `SystemTime` too, I don’t have strong opinion on >> the >> >> name. >> >>>> >> >>>> I proposed `operationTime` because most changelog comes from Database >> >> and we always called an action as `operation` rather than `change` in >> >> Database, the operation time is easier to understand for database >> users, >> >> but it's more like a database terminology. >> >>>> >> >>>> For `SystemTime`, user may confuse which one does the system in >> >> `SystemTime` represents? Flink, Database or CDC tool. Maybe it’s not >> a >> >> good name. >> >>>> >> >>>> `ChangelogTime` is a pretty choice which is more unified with existed >> >> terminology `Changelog` and `ChangelogMode`, so let me use >> `ChangelogTime` >> >> and I’ll update the FLIP. >> >>>> >> >>>> >> >>>>> 2) Exposing it through `org.apache.flink.types.Row`: Shall we also >> >> expose the concept of time through the user-level `Row` type? The FLIP >> does >> >> not mention this explictly. I think we can keep it as an internal >> concept >> >> but I just wanted to ask for clarification. >> >>>> >> >>>> Yes, I want to keep it as an internal concept, we have discussed that >> >> changelog time concept should be the third time concept(the other two >> are >> >> event-time and processing-time). It’s not easy for normal users(or to >> help >> >> normal users) understand the three concepts accurately, and I did not >> find >> >> a big enough scenario that user need to touch the changelog time for >> now, >> >> so I tend to do not expose the concept to users. >> >>>> >> >>>> >> >>>> Best, >> >>>> Leonard >> >>>> >> >>>> >> >>>>> >> >>>>> On 04.08.20 04:58, Leonard Xu wrote: >> >>>>>> Thanks Konstantin, >> >>>>>> Regarding your questions, hope my comments has address your >> questions >> >> and I also add a few explanation in the FLIP. >> >>>>>> Thank you all for the feedback, >> >>>>>> It seems everyone involved in this thread has reached a consensus. >> >>>>>> I will start a vote thread later. >> >>>>>> Best, >> >>>>>> Leonard >> >>>>>>> 在 2020年8月3日，19:35，godfrey he <godfre...@gmail.com <mailto: >> godfre...@gmail.com> <mailto: >> >> godfre...@gmail.com <mailto:godfre...@gmail.com>>> 写道： >> >>>>>>> >> >>>>>>> Thanks Lennard for driving this FLIP. >> >>>>>>> Looks good to me. >> >>>>>>> >> >>>>>>> Best, >> >>>>>>> Godfrey >> >>>>>>> >> >>>>>>> Jark Wu <imj...@gmail.com <mailto:imj...@gmail.com> <mailto: >> imj...@gmail.com <mailto:imj...@gmail.com>>> 于2020年8月3日周一 >> >> 下午12:04写道： >> >>>>>>> >> >>>>>>>> Thanks Leonard for the great FLIP. I think it is in very good >> shape. >> >>>>>>>> +1 to start a vote. >> >>>>>>>> >> >>>>>>>> Best, >> >>>>>>>> Jark >> >>>>>>>> >> >>>>>>>> On Fri, 31 Jul 2020 at 17:56, Fabian Hueske <fhue...@gmail.com >> <mailto:fhue...@gmail.com> >> >> <mailto:fhue...@gmail.com <mailto:fhue...@gmail.com>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Leonard, >> >>>>>>>>> >> >>>>>>>>> Thanks for this FLIP! >> >>>>>>>>> Looks good from my side. >> >>>>>>>>> >> >>>>>>>>> Cheers, Fabian >> >>>>>>>>> >> >>>>>>>>> Am Do., 30. Juli 2020 um 22:15 Uhr schrieb Seth Wiesman < >> >>>>>>>>> sjwies...@gmail.com <mailto:sjwies...@gmail.com> <mailto: >> sjwies...@gmail.com <mailto:sjwies...@gmail.com>> >> >>>>>>>>>> : >> >>>>>>>>> >> >>>>>>>>>> Hi Leondard, >> >>>>>>>>>> >> >>>>>>>>>> Thank you for pushing this, I think the updated syntax looks >> >> really >> >>>>>>>> good >> >>>>>>>>>> and the semantics make sense to me. >> >>>>>>>>>> >> >>>>>>>>>> +1 >> >>>>>>>>>> >> >>>>>>>>>> Seth >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Jul 29, 2020 at 11:36 AM Leonard Xu <xbjt...@gmail.com >> <mailto:xbjt...@gmail.com> >> >> <mailto:xbjt...@gmail.com <mailto:xbjt...@gmail.com>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hi, Konstantin >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> 1) A "Versioned Temporal Table DDL on source" can only be >> >> joined >> >>>>>>>> on >> >>>>>>>>>> the >> >>>>>>>>>>>> PRIMARY KEY attribute, correct? >> >>>>>>>>>>> Yes, the PRIMARY KEY would be join key. >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> 2) Isn't it the time attribute in the ORDER BY clause of the >> >> VIEW >> >>>>>>>>>>> definition that defines >> >>>>>>>>>>>> whether a event-time or processing time temporal table join >> is >> >>>>>>>> used? >> >>>>>>>>>>> >> >>>>>>>>>>> I think event-time or processing-time temporal table join >> >> depends on >> >>>>>>>>> fact >> >>>>>>>>>>> table’s time attribute in temporal join rather than from >> temporal >> >>>>>>>> table >> >>>>>>>>>>> side, the event-time or processing time in temporal table is >> just >> >>>>>>>> used >> >>>>>>>>> to >> >>>>>>>>>>> split the validity period of versioned snapshot of temporal >> >> table. >> >>>>>>>> The >> >>>>>>>>>>> processing time attribute is not necessary for temporal table >> >>>>>>>> without >> >>>>>>>>>>> version, only the primary key is required, the following VIEW >> is >> >> also >> >>>>>>>>>> valid >> >>>>>>>>>>> for temporal table without version. >> >>>>>>>>>>> CREATE VIEW latest_rates AS >> >>>>>>>>>>> SELECT currency, LAST_VALUE(rate) -- only keep the >> >> latest >> >>>>>>>>>>> version >> >>>>>>>>>>> FROM rates >> >>>>>>>>>>> GROUP BY currency; -- inferred >> primary >> >> key >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> 3) A "Versioned Temporal Table DDL on source" is always >> >> versioned >> >>>>>>>> on >> >>>>>>>>>>>> operation_time regardless of the lookup table attribute >> >> (event-time >> >>>>>>>>> or >> >>>>>>>>>>>> processing time attribute), correct? >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Yes, the semantics of `FOR SYSTEM_TIME AS OF o.time` is using >> the >> >>>>>>>>> o.time >> >>>>>>>>>>> value to lookup the version of the temporal table. >> >>>>>>>>>>> For fact table has the processing time attribute, it means >> only >> >>>>>>>> lookup >> >>>>>>>>>> the >> >>>>>>>>>>> latest version of temporal table and we can do some >> optimization >> >> in >> >>>>>>>>>>> implementation like only keep the latest version. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> Best >> >>>>>>>>>>> Leonard >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> > >> > -- >> > Best regards! >> > Rui Li >> >> > > -- > Best regards! > Rui Li >
