Hi Timo, > It would be great if we can add not only `Receive any type of changelog` but > also `Receive any type of datatype`.
Nice, I think we can. > Please clarify whether the compact DDL is a synchronous or asynchrounous > operation in the API? So far all DDL was synchrounous. And only DML > asynchrounous. It should be a synchronous operation. > I find this 'change-tracking' = 'false' a bit confusing. Even in batch > scenarios we have a changelog, only with insert-only changes. Can you > elaborate? Wouldn't 'exclude-from-log-store' or 'exclude-log-store' or > 'log.disabled' be more accurate? Change tracking is from Oracle and snowflake [1][2][3]. It matches the "emit-changes" syntax. It means that after closing, the downstream consumption cannot obtain the corresponding changes. > DESCRIBE DETAIL TABLE +1 to `DESCRIBE TABLE EXTENDED`. > Set checkpoint interval to 1 min if checkpoint is not enabled when the planner detects a sink to built-in dynamic table. This sounds like too much magic to me. You are right. And one minute may not be enough for all situations. +1 to throw detailed exception to alert user. > GenericCatalog to `Catalog#supportesTableStorage` I originally thought about completely distinguishing it from external catalog, but it is also possible to add a new method. > CatalogBaseTable.TableKind Yes, we can create a new TableKind for this table. Catalog can easily distinguish them. > enrichOptions(Context context) Why is this method returning a Map<String, String>? Shouldn't the caller assume that all options enriched via `CatalogTable.copy` should have been applied by `enrichOptions`? Yes, the planner adds some options to the table before creating it. > Partitioning and Event-time: Have you considered to support semantics similar to `sink.partition-commit.trigger` based on `partition-time`. It could beneficial to have the partitions committed by watermarks as well. My biggest concern is how we can enable watermarking end-to-end using a file store (I think for log store this should not be a problem?). Yes, we can also write watermark to storage, which has many advantages: - Users can see the progress of a partition. - Partition commit can have some optimization for this partition, like compact. - Lookup join a table, we can use the watermark to align. Many users complain that the lookup join cannot find data when the lookup table is not ready, it is difficult to align the main stream and the lookup table. But this is not the current blocker. We can improve this in future. [1] https://docs.snowflake.com/en/user-guide/streams.html [2] https://docs.oracle.com/database/121/ADMQS/GUID-3BAA0D48-CA35-4CD7-810E-50C703DC6FEB.htm [3] https://docs.oracle.com/database/121/DWHSG/advmv.htm#DWHSG-GUID-F7394DFE-7CF6-401C-A312-C36603BEB01B Best, Jingsong
